diff --git a/CHANGELOG.md b/CHANGELOG.md index 76ea524a1..a6a8c9a40 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -13,6 +13,32 @@ hashes are given where a change is easiest to understand by reading the commit. ## November 2026 +### PCI + +- The O2's MACE and the Octane's Bridge share one PCI layer (`src/pci/`, + [docs/pci.md](docs/pci.md)): the function table, BAR + decode, interrupt combining, byte order, RST# and a PCI-to-PCI bridge + function (`PciToPciBridge`, not yet fitted to a machine), reached on the + Octane through the Bridge's type 1 configuration aperture. A write that + lowers Octane Bridge reset pin `n` (Control bit `24 + n`) resets PCI device + `n` and logs a warning. Snapshot schema 6: the PCI bus saves its own state, + and older O2 and Octane snapshots are refused. The Octane gains the `pci` + monitor command, and the `pci`, `bridge` and `heart` log modules record + configuration cycles, Bridge register accesses and HEART interrupt vectors + (see [debug.md](debug.md)). +- The Octane's Bridge answers in its widget's 2 GB window at + `0xF_8000_0000`: the first 16 MB is its small window, and from offset + `0x4000_0000` the 32-bit alias of PCI memory, with BARs there mapped + straight to the function as in the device windows. IRIX 6.5.30 reads a + BAR behind a PCI-to-PCI bridge there, which used to be an unmapped access. + +### R10000-family cache lines as memory + +- A valid primary data or secondary cache line now holds its page in that CPU's own copy of ppmem's window, so a PROM that keeps its stack in cache lines reads and writes the lines with cached accesses while the uncached path, DMA and the other CPUs see memory. Index Store Data writes through to such a line. `l2 probe 0` lists the pages held. Nothing changes on a load, store or fetch. A valid R5000 primary data line does the same, for the O2's POST stack. +- Primary instruction lines hold their pages too, each word written there as the instruction it predecodes, for the eleven predecode classes the model knows. Index Load Data (I) where a held secondary line would refill the set returns that line's words, predecoded, until Index Store Data (I) writes the set. Only the Fuel's and Tezro's PROMs use them. +- jitv2 does not compile a page any CPU's cache holds lines in: that CPU runs it from its own view in the interpreter, and every other page stays compiled. +- The O2 keeps POST's stack in those lines. CRIME's side page for the uncached path and the tag-store hook are gone, and an O2 snapshot an older iris took during POST is refused. + ### R10000 family fixes, R12000 and R14000 - The R10000 family has 44-bit virtual and 40-bit physical addresses, FrameMask and a 28-bit EntryLo PFN, picks the TLB refill vector by address space, invalidates overlapped TLB writes (Status.TS), and completes uncached stores nothing answers. Partial LWR keeps the upper word, and the cache diagnostics (Index Store Data, secondary tag mask, Config size) work. @@ -444,6 +470,18 @@ where they differed: - Memory past 4 GB is as fast as the rest: such a machine gets a 16 GB direct window, and the JIT compiles code anywhere in it. - A store-conditional on memory outside the window is now atomic with two CPUs. + +### SGI Fuel (IP34 board, IP35 software platform) + +- **The Fuel takes a VPro board**: `[graphics] board = "v12"` or `"v10"` + fits the Octane's VPro model at crossbar widget 13. IRIX 6.5.30 with the + VPro driver brings the X login screen up on it. + +- Added a headless Fuel profile with the 500 MHz R14000, 512 MiB RAM, + Bedrock, dual-head XBridge, IOC3 networking and two-channel ISP12160 SCSI. + The normal interpreter boot reaches the IRIX 6.5.30 root shell with PROM + 6.210. See the [Fuel guide](docs/fuel-ip34.md). + ### PCI configuration - A PCI function can describe its own writable header fields and diff --git a/Cargo.toml b/Cargo.toml index 37f637eda..bc154616c 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -373,6 +373,14 @@ libc = "0.2" # and CI socket's JSON replies. serde_json = "1.0" +# Model checking of the PCI interrupt combiner's handshake: built only with +# RUSTFLAGS="--cfg loom" (src/pci/intx.rs, `cargo test --lib intx::loom_tests`). +[target.'cfg(loom)'.dev-dependencies] +loom = "0.7" + +[lints.rust] +unexpected_cfgs = { level = "warn", check-cfg = ["cfg(loom)"] } + [profile.release] lto = "fat" codegen-units = 1 diff --git a/FEATURES.md b/FEATURES.md index 9166c7347..5bf7999d2 100644 --- a/FEATURES.md +++ b/FEATURES.md @@ -175,8 +175,8 @@ the first `[section]`, because TOML tables continue until the next header. | Key | Default | Values and behavior | |---|---|---| -| `machine.profile` | `indy_ip24` | indy_ip24, indigo2_ip22, indigo2_ip28, o2_ip32 (configured by `[o2]`, see [the O2 guide](docs/o2-ip32.md)), octane_ip30 (configured by `[octane]`, see [the Octane guide](docs/octane-ip30.md)), origin200_ip29 (configured by `[origin200]`, see [the Origin 200 guide](docs/origin200-ip29.md)), origin2000_ip27 (configured by `[origin2000]`, see [the Origin 2000 guide](docs/origin2000-ip27.md)), onyx2_ip27 (also configured by `[origin2000]`, see [the Onyx2 guide](docs/onyx2-ip27.md)). | -| `machine.cpu` | the machine's first | A family (`r4400`, `r5000`, `r10000`, `r12000`, `r14000`) or a part (`r14000_600`); independent of build features. Indy and `indigo2_ip22`: r4400, r5000; `indigo2_ip28`: r10000; O2: r5000, r10000, r12000; Octane: r10000, r12000, r14000; Origin 200, Origin 2000 and Onyx2: r10000 (`r10000_180`), r12000 (`r12000_270`). The first listed is the default. | +| `machine.profile` | `indy_ip24` | indy_ip24, indigo2_ip22, indigo2_ip28, o2_ip32 (configured by `[o2]`, see [the O2 guide](docs/o2-ip32.md)), octane_ip30 (configured by `[octane]`, see [the Octane guide](docs/octane-ip30.md)), origin200_ip29 (configured by `[origin200]`, see [the Origin 200 guide](docs/origin200-ip29.md)), origin2000_ip27 (configured by `[origin2000]`, see [the Origin 2000 guide](docs/origin2000-ip27.md)), onyx2_ip27 (also configured by `[origin2000]`, see [the Onyx2 guide](docs/onyx2-ip27.md)), fuel_ip34 (see [the Fuel guide](docs/fuel-ip34.md)). | +| `machine.cpu` | the machine's first | A family (`r4400`, `r5000`, `r10000`, `r12000`, `r14000`) or a part (`r14000_600`); independent of build features. Indy and `indigo2_ip22`: r4400, r5000; `indigo2_ip28`: r10000; O2: r5000, r10000, r12000; Octane: r10000, r12000, r14000; Origin 200, Origin 2000 and Onyx2: r10000 (`r10000_180`), r12000 (`r12000_270`); Fuel: r14000 (`r14000_500`). The first listed is the default. | | `machine.cpus` | `1` | Processors fitted. 2 only on the Octane; see [its guide](docs/octane-ip30-multiprocessor.md). | ### [origin2000] @@ -192,11 +192,30 @@ Origin 2000 settings apply when `machine.profile = "origin2000_ip27"` or `"onyx2 An `[origin2000]` table is written to a saved config only when one of these keys is set. See the [Origin 2000 guide](docs/origin2000-ip27.md). +### [fuel] + +Fuel-specific settings apply when `machine.profile = "fuel_ip34"`. + +| Key | Default | Values and behavior | +|---|---|---| +| `flash` | unset | Persistent 16 MiB boot-flash file. Unset, the flash is volatile. | +| `console` | `l1` | `l1`, `ioc3a` or `ioc3b`. Selects the guest console connection. | +| `l1_port` | `8882` | Host TCP port for the L1 console. Used when `console = "l1"`. | +| `l1_log` | unset | Optional serial log path for L1 output when an IOC3 port is selected as console. | +| `l1_revision` | `1.48.1` | L1 firmware revision as `major.minor.patch`, decimal. The guest reads it with L1 request `0x0005`. | +| `brick_serial` | `EMUN00` | L1 brick serial, at most 37 bytes. The guest reads it with L1 request `0x0004`. The default is a marked synthetic serial, not a real one. | + +The machine uses one 500 MHz R14000 and fixed 512 MiB RAM in two 256 MiB +banks. Its IOC3 has no keyboard or mouse ports, so it runs on a serial +console. A `[fuel]` table is written to a saved config only when one of +these keys is set. The sample configuration and supported PROM inputs are in the +[Fuel guide](docs/fuel-ip34.md). + ### [graphics] | Key | Default | Values and behavior | |---|---|---| -| `graphics.board` | `newport`; `solidimpact` on the Octane | newport, xz, extreme, solidimpact, highimpact, maximpact, v6, v8, v10, v12, none. Extreme is Indigo2 only. The Octane takes only `solidimpact` (SI), `highimpact` (SI with texture option), `maximpact` (MXI), `v6`, `v8`, `v10`, `v12` (the VPro boards, also `vpro:v12` and so on; need an IRIX install with the VPro driver, IRIX 6.5.10 or later and PROM 4.0, or 4.5 for the V10 and V12, which shipped only in the Octane2 and so give the machine the Octane2's Crossbow 1.4 and Bridge revision) or `none` (no board: the machine runs on its serial console, with no window). | +| `graphics.board` | `newport`; `solidimpact` on the Octane | newport, xz, extreme, solidimpact, highimpact, maximpact, v6, v8, v10, v12, none. Extreme is Indigo2 only. The Octane takes only `solidimpact` (SI), `highimpact` (SI with texture option), `maximpact` (MXI), `v6`, `v8`, `v10`, `v12` (the VPro boards, also `vpro:v12` and so on; need an IRIX install with the VPro driver, IRIX 6.5.10 or later and PROM 4.0, or 4.5 for the V10 and V12, which shipped only in the Octane2 and so give the machine the Octane2's Crossbow 1.4 and Bridge revision) or `none` (no board: the machine runs on its serial console, with no window). The Fuel takes `v12` or `v10` in its graphics slot, or `none`, its default. | | `graphics.heads` | `1` | 1 or 2; dual-head requires Newport. | | `graphics.resolution` | `guest` | guest, 1024x768, 1280x960, 1280x1024; presets require Newport. | | `graphics.threads` | `2` | Worker threads the board's fragment pipeline draws on, at least 1. VPro: the screen tiles' owners, besides its front-end and raster threads; more help on a host with cores to spare. | @@ -231,7 +250,7 @@ An explicit non-disk kind takes precedence over the legacy cdrom flag. | Key | Default | Values and behavior | |---|---|---| -| `network.mode` | `nat` | nat, pcap, stream or off (no Ethernet; O2 and Octane only). Stream needs `network.stream`. PCAP needs build support and host capture permissions. PCAP works on every Ethernet; stream on every onboard one, not on a DaynaPort. | +| `network.mode` | `nat` | nat, pcap, stream or off (no Ethernet; O2, Octane and Fuel only). Stream needs `network.stream`. PCAP needs build support and host capture permissions. PCAP works on every Ethernet; stream on every onboard one, not on a DaynaPort. | | `network.stream` | `unset` | `tcp:IP:PORT` or `unix:PATH`: the server of the private segment for `mode = "stream"` (QEMU `-netdev stream` framing). See NETWORKING.md. | | `network.pcap_interface` | `auto-pick when unset` | Interface name or numeric list index; PCAP only. | | `network.nfs_pcap_ip` | `unset` | Virtual LAN IP for the built-in NFS responder in PCAP mode. | @@ -395,8 +414,8 @@ IP22/IP24 four-bank IRIX 6.5 behavior remains separate from IP28's verified two-bank 1 GB layout. Extreme requires IP22. All non-Newport boards require `heads=1` and `resolution="guest"`. IP28 IRIX needs IMPACT; its PROM has an embedded fallback. -The config validator permits SCSI controller 1 only on IP22, the O2 and the -Octane, the machines with two controllers. +The config validator permits SCSI controller 1 only on IP22, the O2, Octane +and Fuel. These machines have two controllers. Standalone CLI options update the loaded config where supplied; boolean enable switches do not provide a general way to clear every saved true setting. @@ -449,7 +468,7 @@ iris-gui does not parse this standalone option set; use its named-machine UI. | `--list-net-interfaces` | Print the host network interfaces libpcap can bridge onto, then exit. Requires a build with --features pcap. | | `--no-scsi-deferred-int` | Disable deferred SCSI status interrupts (default: enabled for OpenBSD/NetBSD compatibility). | | `--cpu` | Emulated CPU: a family (`r4400`, `r5000`, `r10000`, `r12000`, `r14000`) or a part (`r14000_600`) the machine takes. Overrides `[machine] cpu`. | -| `--graphics` | Graphics board: `newport` (default), `xz`, `extreme`, `solidimpact`, `highimpact`, `maximpact` (or `impact:solid`, `impact:high`, `impact:max`), `v6`, `v8`, `v10`, `v12` (`vpro:v12` and so on, Octane only). | +| `--graphics` | Graphics board: `newport` (default), `xz`, `extreme`, `solidimpact`, `highimpact`, `maximpact` (or `impact:solid`, `impact:high`, `impact:max`), `v6`, `v8`, `v10`, `v12` (`vpro:v12` and so on, Octane; `v10`/`v12` on the Fuel). | | `--raster` | VPro and IMPACT: `cpu` (default) or `wgpu`, the raster unit on the host GPU (`[graphics] raster`; needs a build with the `wgpu` feature). | | `--renderer` | What draws the window: `gl` (default) or `wgpu` (`renderer`; needs a build with the `wgpu` feature). | | `--gdb-port` | Enable GDB stub on the given TCP port (e.g. --gdb-port 1234). Connect with: target remote localhost: | diff --git a/HACKING.md b/HACKING.md index 52d123c0d..5d138bc07 100644 --- a/HACKING.md +++ b/HACKING.md @@ -16,13 +16,18 @@ Other documents: ## 1. What IRIS is -IRIS emulates five SGI workstations: the Indy (IP24), the Indigo2 (IP22), the -Indigo2 IMPACT (IP28), the O2 (IP32) and the Octane (IP30). The CPU is an -R4400, R5000, R10000, R12000 or R14000, chosen at runtime from the parts the -machine shipped with (`src/cpu/parts.rs`). -Graphics is Newport (REX3), GR2 (XZ/Extreme) or IMPACT on the Indy and -Indigo2, GBE on the O2 and Solid IMPACT on the Octane. It boots the real PROMs -and IRIX 5.3 and 6.5 to a desktop with networking. +IRIS emulates the Indy (IP24), Indigo2 (IP22), Indigo2 IMPACT (IP28), O2 +(IP32), Octane (IP30) and Fuel (IP34 board, IP35 software platform). The CPU +is an R4400, R5000, R10000, R12000 or R14000, chosen at runtime from the +parts the machine shipped with (`src/cpu/parts.rs`); the Fuel has its +500 MHz R14000. Graphics is Newport +(REX3), GR2 (XZ/Extreme) or IMPACT on the Indy and Indigo2, GBE on the O2, and +Solid IMPACT on the Octane. Fuel has no graphics board and runs headless on +its serial console. It does have a PS/2 keyboard and mouse on its IOC3, fed +from the host's input and the default as in the Fuel's manual, and `[fuel] +usb_input = true` adds a USB keyboard and mouse on its OHCI controller; +`ps2_input = false` turns PS/2 off. The graphical machines boot their real +PROMs and IRIX 5.3 or 6.5 to a desktop with networking. It is **not cycle-accurate**, anywhere. IRIX is interrupt-driven and does not care how many cycles an instruction takes, so IRIS models behaviour @@ -83,17 +88,18 @@ image. A CD-ROM's node is a `ChangerNode`: the queue of discs and the open active one, with the `Removable` trait (present, changed, eject, load) separate from `BlockNode` so that removable media which is not a block device, a tape, can use it too. The WD33C93A (Indy, Indigo2), the Adaptec -7880 (O2) and the ISP1040 (Octane) each make an empty bus of their own kind -(`new_bus`); the machine fills it from config and gives it to the controller's -constructor, so a controller exists with its targets and keeps only its -registers and DMA. What a `BlockNode` must guarantee about durable writes, -and why a disk's `write_cache` setting matters, is in the "Disk write cache" -section of [STORAGE.md](STORAGE.md) and the `block_node.rs` module docs: read -them before writing a backend or changing a flush. `src/scsi_monitor.rs` implements the monitor's `scsi` and `cow` +7880 (O2) and ISP1040 (Octane) each make an empty bus of their own kind +(`new_bus`). Fuel uses an ISP12160 with two SCSI buses behind one PCI +function. The machine fills each bus from config and gives it to the controller +constructor, so the controller keeps its registers and DMA. What a `BlockNode` +must guarantee about durable writes, and why a disk's `write_cache` setting +matters, is in the "Disk write cache" section of [STORAGE.md](STORAGE.md) and +the `block_node.rs` module docs: read them before writing a backend or changing +a flush. `src/scsi_monitor.rs` implements the monitor's `scsi` and `cow` subcommands and the run-time disc changes once, over the bus a controller hands -out through `ScsiTargets::scsi_bus`; chip-specific subcommands (`regs`; the -WD's `wdt`, `defer`, `debug`) stay with the chip, and `DiscChanger` is -implemented once for every `ScsiTargets`. Configuration is shared the same +out through `ScsiTargets::scsi_bus`. Chip-specific subcommands stay with the +chip, including `regs` and the WD's `wdt`, `defer` and `debug`. `DiscChanger` +is implemented once for every `ScsiTargets`. Configuration is shared the same way: `scsi::attach_from_config` walks the `[scsi.N]` tables for every machine, where the key is a slot (controller × 16 + id, see `config::scsi_slot`), and the machine supplies only a route from controller number to bus. @@ -101,6 +107,13 @@ the machine supplies only a route from controller number to bus. Flash command engines share `FlashBacking` file storage while retaining their own command state. +`ScsiTarget::bus_reset` defaults to no action. Disk targets record reset as +unit attention. Only the ISP12160 calls it; the other controllers keep +their earlier behaviour of not reporting resets to targets. +The ISP12160 defers target resets and releases its controller state lock +before walking the bus. It calls targets while holding the bus lock, so a +target callback must not take that lock again. + ## 2. Data path and endianness **Word-transparent architecture.** Host `u32`/`u64` values are bit @@ -198,16 +211,17 @@ view of it); `physical::read_bytes`/`write_bytes` move byte ranges a page at a time with `read_block`/`write_block`, and both PCI host bridges' `PciDma` use them. -PCI is shaped the same way on both machines that have it. A host bridge -(the Octane's Bridge, the O2's MACE) works out which pages of its CPU -windows one BAR decodes whole (`pci::bar_runs`) and hands them to the -machine's bus through a callback (`on_bar_runs`); the bus maps each run on -`Physical` as a `pci::PciBar` entry, which drives the function directly -through the bridge's byte-lane rules (`pci::PciLanes`). The bridge -recomputes the runs after a configuration write, a write to a window -register and a reset. Everything else in a window (BARs smaller than a -page, shared pages, unclaimed addresses and their master aborts) stays with -the bridge's own decode. +PCI is shaped the same way on both machines that have it +([docs/pci.md](docs/pci.md)). A host bridge (the Octane's Bridge, the O2's MACE) +owns a `pci::PciBus`, which works out which pages of the host's CPU windows one +BAR decodes whole and hands them to the machine's bus through a callback +(`PciBus::on_bar_mappings`); the bus maps each run on `Physical` as a +`pci::BarDevice` entry, which drives the function directly through the window's +byte order (`pci::ByteOrder`, held by value). The bus updates the BAR mappings +after a configuration write, and the host after a write to a window register. +Everything else in a window (BARs smaller than a page, shared pages, unclaimed +addresses and their master aborts) stays with the host's own decode, which asks +`PciBus::decode`. `Physical` decodes physical addresses through a table of 4 KB slots (one `*const dyn BusDevice` per `addr >> 12`), so finding the @@ -304,9 +318,12 @@ Every device that has real work to do runs on its own thread: | `VNC-Grab`, `VNC-Pointer`, `VNC-Client`/`VNC-Read` | with `--vnc`: frame diffing, pointer steering, one pair per viewer | | `machine-events` | power-off and reset requests from devices | | `ISP1040-0`, `ISP1040-1` | Octane SCSI request rings and disk transfers | -| `IOC3-RX-A`, `IOC3-RX-B`, `IOC3-NVRAM` | Octane serial input per port; the DS1685 NVRAM file writer (IOC3's other deadlines are `hptimer` one-shots) | +| `ISP12160-0` | Fuel SCSI request ring for both ISP12160 channels | +| `IOC3-RX-A`, `IOC3-RX-B`, `IOC3-NVRAM` | Octane and Fuel IOC3 serial input and NVRAM writer (IOC3's other deadlines are `hptimer` one-shots) | +| `l1-serial` | Fuel L1 console UART worker. Its other serial ports are IOC3 A and B | | `ip30-eth-rx`, `ip30-eth-net` | Octane ethernet receive pump; the network backend | | `odyssey-work`, `odyssey-raster`, `odyssey-pool-N`, `odyssey-display` | VPro: the front end decodes the queued command-FIFO stores (the processor's stores only queue); the raster unit sets each draw up and walks its primitives into 2x2 quads; a pool of workers (`[graphics] threads`, default 2) walks, shades and stores filled triangles and stores every other fragment, one screen tile at a time; and the display composes each whole frame from board memory (64-bit words, as main memory is: a field is a bit range of its word, read by shift and mask with no byte order, `vram.rs`) as the scan-out registers describe it, counts the retrace and presents the window at 60 Hz. No board-wide lock: the front end owns its state, an access that needs the drawing is answered busy until it is done, the display FIFO (PB&J's register file) has its own lock and never waits for the drawing (`display_fifo.rs`), and the display reads board memory while it is drawn, as the board's own scan-out does; nobody tracks which rows changed. The raster thread sets primitives up and writes each as one command in a ring of 64-byte slots (`pipe/ring.rs`): a triangle's record and bounding box, a quad it shaded itself (lines, points), or a clear's rectangle; the slot's index goes on the queue of every screen tile (`[graphics] tile`, default 32 pixels) it touches. A worker takes a ready tile and owns it until its queue is empty, walking each triangle within the tile and shading and storing quad by quad, so every pixel sees its fragments in command order with no other ordering, and the depth pre-test reads depth its own tile stored. A slot is retired once all its tiles have run it. The raster thread drains the ring before any other work on board memory (a texture upload waits for the shader alone). A completion (a fence's token, a query's words) does not drain: it rides the ring and runs once the ring retires past the quads queued before it. A quad's four pixels are lanes: the walk's depth, the texture unit's positions and the colour weights run on the host's SIMD unit (`fearless_simd`, `pipe/lanes.rs`, the level detected at start-up), bit for bit the scalar arithmetic. When its queue runs dry the raster thread does not drain: it keeps one count of the raster unit's pending jobs for the quads still in flight, and a front-end wait or a refused processor read asks it to store them. `IRIS_ODSY_SHADER=inline` makes the raster thread shade every quad itself, and `IRIS_ODSY_BACKEND=inline` stores on the raster thread with no pool. The texture registers a draw reads live in an array of draw state slots (`pipe/slot.rs`): the front end writes the current slot, a draw latches it and holds it until its job is given back, and a write while a queued draw holds the current slot first copies it to a free one, so the stages read their draw's registers with no lock | +| `ioc3-eth-rx`, `ioc3-eth-net` | Fuel IOC3 Ethernet receive pump and network backend | Threads that have nothing to do park instead of spinning: the graphics consumers when their FIFO is empty, and the CPU thread in `idle-pause` builds @@ -746,6 +763,15 @@ introspection commands are in [debug.md](debug.md#jit-v2). Crossbow and the Bridge with IOC3 and two ISP1040 SCSI controllers, an R10000, and IMPACT graphics (see the [Octane guide](docs/octane-ip30.md) for the boards). +- **IP34 board, IP35 software platform (Fuel)**: one 500 MHz R14000, 512 MiB + in two 256 MiB banks, Bedrock, two XBridge heads, IOC3 and a two-channel + ISP12160. It runs headless on the L1 or IOC3 serial console. The IOC3 uses + the shared `net::Link` for NAT or PCAP. The writable Bedrock flash and + serial-flash devices share `FlashBacking` storage, with separate command + handling. A VPro board (the Octane's Odyssey model) can sit at crossbar + widget 13; its packets and DMA go through the XBridge and Bedrock's inbound + path, and the PI graphics registers are stored but count no credits. + Snapshots are unsupported. See the [Fuel guide](docs/fuel-ip34.md). The Indy and Indigo2 profiles each have an embedded PROM fallback; `src/platform.rs` and `src/machine.rs` wire their differences, and diff --git a/HELP.md b/HELP.md index f7208eaf5..77ace5234 100644 --- a/HELP.md +++ b/HELP.md @@ -1,7 +1,9 @@ -# IRIS — SGI Indy and Indigo2 emulator +# IRIS — SGI workstation emulator The complete core/GUI feature and configuration inventory is in -[FEATURES.md](FEATURES.md). This guide explains everyday use; the monitor's full command reference is in [debug.md](debug.md). +[FEATURES.md](FEATURES.md). For the Fuel, see the [Fuel guide](docs/fuel-ip34.md). +This guide explains everyday use. The monitor's full command reference is in +[debug.md](debug.md). ## Quick start @@ -905,6 +907,7 @@ likely to need while just using IRIS: | `rtc save` / `nveeprom save` | Write the Indy NVRAM / Indigo2 EEPROM to disk now | | `scsi status` / `scsi eject ` / `scsi add ` | Show disks and CD queues, change CDs | | `ps2 type ` / `ps2 enter` / `ps2 esc` | Type on the guest keyboard (Esc stops PROM autoboot) | +| `pci status` / `pci config ` | O2 and Octane: the functions on the PCI bus / one function's configuration header | | `net status` | NAT connections | | `mc regs` | Memory configuration, including MEMCFG | diff --git a/clanker/INDEX.md b/clanker/INDEX.md index b64133f49..f5f0ad783 100644 --- a/clanker/INDEX.md +++ b/clanker/INDEX.md @@ -24,6 +24,7 @@ CPU architecture: CP0, TLB and its fast paths, caches, FPU, LL/SC, CPU models. B | Note | Kind | Title | |---|---|---| | [cached-flash-is-a-cache-with-two-cpus.md](cpu/cached-flash-is-a-cache-with-two-cpus.md) | rule | With two CPUs, cached flash reads must hold their lines: a fetch through to a chip in status mode executes `0x80808080` | +| [cache-lines-live-in-the-cpus-view.md](cpu/cache-lines-live-in-the-cpus-view.md) | design | A valid R10000-family cache line, or R5000 primary data line, makes its page private in that CPU's own copy of ppmem's window; the uncached path and other masters keep memory. Limits and how to see what is held | | [cache-attributes-and-fetch-alignment.md](cpu/cache-attributes-and-fetch-alignment.md) | rule | Cache attributes and fetch alignment: three hardcodings, all wrong | | [cache-tag-index-and-check-bits.md](cpu/cache-tag-index-and-check-bits.md) | rule | Cache tags: index ops index by address; check bits are generated, not stored | | [cop0-requires-kernel-or-cu0.md](cpu/cop0-requires-kernel-or-cu0.md) | rule | CP0 instructions must be gated on Kernel-or-CU0 | @@ -100,6 +101,8 @@ Non-graphics devices: MC/MEMCFG and RAM banks, HPC3, VDMA, EEPROMs and NVRAM, PS | Note | Kind | Title | |---|---|---| +| [pci-layer-design.md](devices/pci-layer-design.md) | design | The PCI layer: headers change only through the bus, byte orders are per-window values, `pci/` imports no card or SCSI; the stale-run and delivery-order hazards, the SeqCst handshake, and what the first attempt got wrong | +| [pci-layer-open.md](devices/pci-layer-open.md) | open | PCI layer open items: restore vs reset and the AIC-7880 chip reset, the O2's RST#, Bridge reset pins and master time-out approximations, the 2 GB window's reserved range, no external ATE SSRAM, MACE 64-bit PIO | | [virtio-serial-design.md](devices/virtio-serial-design.md) | design | virtio-serial: one transport-neutral core, mmio in a GIO slot (IP22/24/28) and pci (IP30); threading, byte order and open items | | [empty-gio-slot-reads-bus-error.md](devices/empty-gio-slot-reads-bus-error.md) | rule | Reads of an empty GIO slot end in a bus error (ABORT_EN); all ones made the PROM poll a phantom Newport 200 000 times headless | | [audio-test-suite.md](devices/audio-test-suite.md) | howto | The audio test suite (`test/audiotest`): rates x channels x widths through the audio library, playaiff and sfplay, checked against the Octane's capture | @@ -126,7 +129,7 @@ Non-graphics devices: MC/MEMCFG and RAM banks, HPC3, VDMA, EEPROMs and NVRAM, PS | [aic7880-patchsim.py](devices/aic7880-patchsim.py) | howto | Reproduces the Linux aic7xxx firmware patch selection and critical sections for a 7880 | | [o2-no-ecc-alias-is-memory.md](devices/o2-no-ecc-alias-is-memory.md) | rule | O2: physical 0x00_8000_0000 is CRIME's no-ECC memory alias, folded onto RAM by the CPU (netstat/Netscape hang, /dev/mem all ones) | | [o2-cache-handover-full-to-shadow.md](devices/o2-cache-handover-full-to-shadow.md) | history | O2 cache as it was until 2026-10-30: the full model through POST, the shadow model after, switched by PC range and a tag-store trigger | -| [o2-prom-stack-is-a-cached-page-at-phys-0.md](devices/o2-prom-stack-is-a-cached-page-at-phys-0.md) | design | O2 PROM stack: one tag-installed page at physical 0 that the memory test overwrites through KSEG1; a 4 KB page in the bus decode answers for it uncached, so the shadow cache runs from reset | +| [o2-prom-stack-is-a-cached-page-at-phys-0.md](devices/o2-prom-stack-is-a-cached-page-at-phys-0.md) | design | O2 PROM stack: one tag-installed page at physical 0 that the memory test overwrites through KSEG1. Retired: the CPU's cached view holds it now | | [memcfg-limit-is-not-the-bank-size.md](devices/memcfg-limit-is-not-the-bank-size.md) | rule | MEMCFG: `limit` is the slot, `addr_mask+1` is the mirror period — they differ | | [nvram-persistence.md](devices/nvram-persistence.md) | design | Battery-backed state on Stop | | [ps2-aux-mux-probe-and-status-bit.md](devices/ps2-aux-mux-probe-and-status-bit.md) | rule | PS/2 mouse not detected under Linux 3.2 guests (and keyboard sometimes flaky too) | @@ -465,6 +468,7 @@ Plans, task briefs and reviews written for agents. Each states its own status; m | [odyssey-quad-pipeline.md](plans/odyssey-quad-pipeline.md) | plan | Plan: a three-stage quad pipeline for the VPro rasteriser (rasteriser, shader, back end on threads; quake1 timedemo as the workload) | | [wgpu-raster.md](plans/wgpu-raster.md) | plan | Plan: the boards' raster units on the host GPU through wgpu compute (board memory on the GPU, tile-binned batches; common layer landed, VPro first, IMPACT second (raster unit landed, in the board behind `raster = "wgpu"`), GR2 open) | | [odyssey-board-memory.md](plans/odyssey-board-memory.md) | plan | VPro board memory as the hardware has it: one tiled SDRAM pool addressed by base/pitch/format, replacing the fixed planes and the invented object store; measure the tile layout on a real V10 first | +| [pci.md](plans/pci.md) | plan | PCI: what the shared layer delivers, then the Fuel and Tezro conversions, the XIO PCI carrier, card cage and NVMe in real slots, and the post-rework reset work | | [pci-dma-port.md](plans/pci-dma-port.md) | plan | PCI bus masters: one `PciDma` trait, shared on the O2, per device on the Octane's Bridge; Bridge/Xbridge DMA, PIO, interrupt and swap facts; step 1 done | | [o2-memory-and-locks.md](plans/o2-memory-and-locks.md) | plan | O2 disk I/O gap vs Indy: interrupt re-drive on every access (~205 locks), PCI BAR decode locks, debug logs on hot paths, RAM onto Physical/SparseDeviceMap/PpMemory like the Indy, remaining locks; hardware facts on memory masters | | [networking-tab-redesign.md](plans/networking-tab-redesign.md) | plan | Networking tab redesign — design & plan | diff --git a/clanker/cpu/cache-lines-live-in-the-cpus-view.md b/clanker/cpu/cache-lines-live-in-the-cpus-view.md new file mode 100644 index 000000000..a2830778d --- /dev/null +++ b/clanker/cpu/cache-lines-live-in-the-cpus-view.md @@ -0,0 +1,106 @@ +# Cache lines live in the CPU's own view of memory + +The R10000-family PROMs use the caches as memory before memory works. They make +lines valid with Index Store Tag and Index Store Data and then load, store and +fetch through them, while uncached accesses, which bypass the caches (NEC +VR10000 user's manual p. 75), and every other master see memory. What the shipped PROMs do on iris: + +| Machine | Lines | Memory behind them | +|---|---|---| +| O2 | secondary and primary data lines for page 0 and 0x1000 hold POST's stack for about 50-65 ms, while POST tests page 0 uncached. On the R5000, primary data lines for page 0 (TagLo 0xc0) | yes | +| Octane | a tag and data RAM test on secondary set 0, read back only with Index Load Tag/Data | no: the patterns name no memory | +| Fuel | primary data lines for 0x100000-0x107fff hold the stack; primary instruction lines hold PROM code | no, until memory is configured mid-phase | +| Tezro | the same as the Fuel, on every processor at once, with different stack contents at the same addresses | the same | + +Every line ends by Index Store Tag. None is written back, and none is read back +with Index Load Data while it holds a page. + +## How + +- Each CPU reads memory through a window of its own (`PpMemSpace::add_cached_view`). + Every mapping laid in the shared window is laid in each view too, and the view's + region bitmap goes to that CPU's inline fields. The shared window, which + `Physical`'s direct paths, CRIME's rendering engine and HEART read, never holds a + line. +- When Index Store Tag makes a line valid, its page becomes private memory in that + view, a copy of memory, and the line's data array is written into it + (`ShadowCache::line_enters`). An invalid or different tag ends the line, and the + page goes with its last line. The shadow's tags stay the only line state, with a + held bit beside each. Index Store Data into a held line writes through. +- The R5000 does the same for a valid primary data line, whose tag has the + R10000's PTag and PState fields (NEC VR5000 user's manual U11761EJ6V0UM p. 133). + It has no Index Store Data, so the page starts as memory. Its instruction lines + hold nothing, since one could hold only what memory holds. +- A page with no memory behind it holds nothing: the line stays in the arrays, as + a tag test's patterns do. The region bitmap is all a CPU tests, so a held page's + whole region is the view's while it stands, reading zeros wherever nothing is + mapped. The O2's POST holds lines for page 0 before CRIME maps the rest of that + region. +- A remap saves and restores held pages, so memory configuration mid-phase keeps + the stack. +- Loads, stores, fetches, the decode cache and compiled code are untouched: they + read the view through the base pointer and bitmap they always had. The decode + cache compares every word it serves, so it sees a page change hands. + +## Limits + +Each is an approximation, stated where it is made. + +- Page, not line: the rest of a held page shows memory as it was when the page was + taken, uncached writes to it meanwhile are not seen by cached reads, cached stores + to it are dropped at release, and a line that ends leaves its bytes until the + page's last line goes. The page is the host granule: 4 KB on Linux and x86 macOS, + 16 KB on Apple silicon, 64 KB on Windows. +- Lines end only by a tag change the shadow already models: Index Store Tag, and + Index Writeback Invalidate (D) zeroing the tag from the R12000 on. Nothing is + written back. The hit operations and the other index invalidates leave a held + line as they leave every shadow tag, untouched. +- Index Load Data reads the data array, not the page, so it does not see cached + stores. +- A region holding a page reads zeros in its unmapped gaps for that CPU's cached + accesses, where the bus would answer. +- A secondary doubleword puts TagLo at the lower address. The manuals say only + that TagHi is the high word and TagLo the low word (MIPS R10000 UM p. 201, NEC + VR10000 UM p. 289); which address each pairs with is inferred. PROM 6.210 (Fuel, + Tezro) stages code with the word at A in TagLo and the word at A + 4 in TagHi, + 990 times in a Tezro boot, and its staged code equals its image only this way. + The IP35 boot-flash PROM's staging pairs the words the same way: simulated with + the other order, 378 of its 562 staged fetches break, every trampoline fetch + among them. With the other order the Tezro's PROM ran its long cache test + uncached, taking twice as long. +- No replacement, parity or ECC, and no coherence with other CPUs or DMA. +- A snapshot taken while lines hold pages is refused at restore. + +## Compiled code + +A page some CPU holds is not compiled: compiles read memory, and compiled code is +shared. ppmem puts `GEN_HELD` on the page's generation counter at the first hold +and takes it off at the last release, moving it with the page on a remap +(`PpMemSpace::mark`). The compile that finds it publishes nothing and denies every +offset until the release. ppmem's counters are the bus's for memory. Where the bus +maps a device over memory, as the IP35 boards map their reset PROM, the compile checks +that device's counter instead, so the cache puts the same mark there when a page +becomes held and takes it off with the page's last line (`mark_foreign`). That work +is at CACHE instructions only; a load, store or fetch is unchanged. + +## Primary instruction lines + +Only the Fuel's and the Tezro's PROMs use these. A valid primary instruction line +writes each 36-bit predecoded word (p. 288) into its page as the instruction it +predecodes, through an eleven-class table inferred from PROM 6.210's words +(`l1i_decode`). A word of another class leaves memory's word, where hardware would +run it. A page held only for code also answers cached loads with code. + +The model never refills on a fetch. The Tezro's PROM runs a staged secondary line +and reads the primary words back with Index Load Data (I) to stage them itself, so +Index Load Data (I) in a set a held secondary line maps to answers with that line's +words predecoded (`refilled_word`), until Index Store Data (I) writes the set. When +two held lines map one set, the later wins. An instruction no class holds comes back +in the model's own form (bits 35:32 set, which takes classes 0x78-0x7f), which +hardware would not produce. Index Load Tag (I) reads the stored tag. + +## Seeing it + +`l2 probe 0` in the monitor prints how many lines hold pages and which. Every +machine should show none at the IRIX login. `log l2c mask op` logs each page held +and released. diff --git a/clanker/devices/aic7880-emulation-vs-spec.md b/clanker/devices/aic7880-emulation-vs-spec.md index 7b8b9ecd9..f877cd35d 100644 --- a/clanker/devices/aic7880-emulation-vs-spec.md +++ b/clanker/devices/aic7880-emulation-vs-spec.md @@ -201,7 +201,7 @@ design; the rest are register-level details that could be fixed inside it. drivers check `INTSTAT` first, so this is a safe simplification, but note it when implementing stepping (`SEQCTL.STEP` is ignored today). -## C. PCI configuration space (`src/pci.rs` as used by `Aic7880::new_at`) +## C. PCI configuration space (`src/pci/config.rs` as used by `Aic7880::new_at`) 23. **Expansion ROM BAR (`0x30`) stores any value** (§5). The chip implements a 64 KB ROM BAR (bits 31:15 writable, 14:11 read 0, bit 0 diff --git a/clanker/devices/o2-prom-stack-is-a-cached-page-at-phys-0.md b/clanker/devices/o2-prom-stack-is-a-cached-page-at-phys-0.md index 5356cce38..39f6e935b 100644 --- a/clanker/devices/o2-prom-stack-is-a-cached-page-at-phys-0.md +++ b/clanker/devices/o2-prom-stack-is-a-cached-page-at-phys-0.md @@ -1,10 +1,11 @@ # O2 PROM stack is one cached page at physical 0, and RAM behind it is a different thing -Status: landed 2026-10-30. Both O2 CPUs run the shadow cache from reset; the handover -([o2-cache-handover-full-to-shadow.md](o2-cache-handover-full-to-shadow.md)) is gone. Code: -`O2Memory::stack_tag_store` in `src/dev/crime_mem.rs`, the tag-store hook in -`src/cpu/mips_cache_shadow.rs`. Sections below are the measurements that chose the design; the -prototype results used earlier binaries. +Status: retired 2026-11-08. The stack now lives in the CPU's own cached view of ppmem's window: +a valid primary data or secondary line holds its page there (`ShadowCache::line_enters` in +`src/cpu/mips_cache_shadow.rs`, `PpMemSpace::add_cached_view`), and the uncached path, the decode +table and the rendering engine keep seeing memory. `O2Memory::stack_tag_store`, its side page and +the tag-store hook are gone. The sections below are the measurements that chose the old design; +what they say about the PROM still holds. ## Where the stack is diff --git a/clanker/devices/pci-layer-design.md b/clanker/devices/pci-layer-design.md new file mode 100644 index 000000000..62ad7f18b --- /dev/null +++ b/clanker/devices/pci-layer-design.md @@ -0,0 +1,55 @@ +# The PCI layer: invariants, hazards, and what v1 got wrong (design) + +Status: in place for MACE and the Bridge, 2026-11. The user-facing description +is `docs/pci.md`. + +## Invariants + +- **Headers change only through the bus** (`config_write`, `reset`, snapshot + load). Each updates the decode table and the BAR mappings. A function that + resets its own header leaves a stale BAR mapping on the physical bus. +- **Byte orders are per-window values**, `Copy`, chosen by the host and held by + value in each `BarDevice`. The bus never sees byte order. +- **`pci/` imports no card, no SCSI and no host.** The one thing it takes from + outside is the physical bus's page size for the run logic. +- **Snapshot restore is not RST#.** It loads over what is there and ends + with `PciBus::redrive`. A function's load must overwrite what the + snapshot does not name (the AIC-7880 puts its SCSI targets back with + `reset_targets`). +- **The combiner's handshake is one atomic word per output** (`DIRTY`, + `HELD`, `FELL`, `FORCE`). Every step is a read-modify-write on that word, so + acquire-release is enough; a store-then-load handshake would need SeqCst and + cannot be model-checked. `loom_tests` in `src/pci/intx.rs` checks it + (`RUSTFLAGS="--cfg loom"`). + +## Hazards + +- A stale run table: anything that changes a BAR, a command register, a bridge + window or a host window must update the mappings. A change behind a bridge + updates the buses in front of it. A configuration write to any other register + does not, because each update walks the whole table. +- The physical bus keeps every device it is given, for the life of the + machine. Hosts make their `BarDevice`s through `BarDevices`, which returns the + same device for the same function, BAR, base and order. +- Delivery ordering: the host's own lock must not be taken by anything that + calls into the combiner with that lock held. The Bridge decides a packet + under `st` and sends it after dropping `st`. +- Lock order: publish mutex, then the host's lock (`refresh`'s `make`). The + Bridge drops `st` before `refresh`. A bridge's secondary holds its parent + weakly; the parent holds the bridge function, which holds the bus. +- The slow BAR table is scanned without a lock while an update rewrites it. A + reader can briefly see a mix of old and new regions; every word is valid and + no slot is freed, so it is not unsafe. + +## What v1 (`pci-addin`) got wrong + +- Cards, SCSI and the snapshot format leaked into the core (`CardRef`, + `Origin`, `attach_card`, `ScsiController`). +- Three copies of the level state in the Bridge and a lock held across the + host call in MACE's router. +- A byte-order trait object per host and a duplicate swizzle in each. +- Reset was whatever each function's power-on did, and the O2 had none. + +## Open + +See [pci-layer-open.md](pci-layer-open.md). diff --git a/clanker/devices/pci-layer-open.md b/clanker/devices/pci-layer-open.md new file mode 100644 index 000000000..9cd8dc131 --- /dev/null +++ b/clanker/devices/pci-layer-open.md @@ -0,0 +1,54 @@ +# PCI layer: open items + +Status: open, 2026-11. + +- **O2 RST# on cold reset** is not wired yet; the O2's cold reset leaves its + PCI functions alone. When it is, it is inferred. MACE has a `PCIRst_N` output and + CRIME drives the system reset, but no document says the board connects them, + and no MACE register resets the PCI bus (MACE specification pp. 123, 162-165, + 188, 194). The O2 Device Driver Programming Guide (007-3443-002) mentions + only the generic `pciio_reset()` interface; what that does on an O2 is + unknown. +- **Restore is not reset, and the AIC-7880 chip reset.** A snapshot load puts + the functions back over what is there and ends with `PciBus::redrive`; it + does not pulse RST#. The AIC-7880 has no `reset` of its own: RST# clears + its header and nothing else, and its chip reset is the `HCNTRL` `CHIPRST` + write. Its load calls `reset_targets`, which is the SCSI bus reset + (SCSIRSTO), to put the targets where a bus reset leaves them. Telling RST#, + chip reset and SCSI bus reset apart, and making a load re-create rather + than reset, is deferred until after the machine construction is reworked. +- **Bridge Control 27:24**: pin `n` resets PCI device `n` alone. The + pin-to-device wiring is inferred from `pciio_reset()` ("without affecting + other devices on the PCI bus", p. 550) and the four-pin count; the Bridge + specification does not give it. Whether the stock PROM or IRIX ever writes a + pin low has not been observed. A write is logged as a warning. +- **Bridge master time-out capture is an approximation.** Which device number + the PCI error upper register names for a type 1 cycle nobody claims is not + in the Bridge specification; iris records the device the cycle carried. For + a PIO through the 2 GB window's PCI memory alias that nothing claims, iris + names device 0 and records the PCI address the cycle carried. +- **Bridge 2 GB window, 16 MB to 1 GB, is an approximation.** The + specification calls it reserved (p. 52) without saying what an access does; + iris answers with a bus error. The 64-bit alias and PCI I/O space, + reachable only through the 64 GB window, are not modelled. +- **No external ATE SSRAM.** The Octane does not fit it; `0x080000` reads as + all ones, the way the PROM's size probe expects, and a write is accepted + and dropped. +- **MACE 64-bit PIO** is not in the driver guide's table; iris splits it into + two word cycles, high word first. MACE rev 0 sends 4-byte-aligned addresses + on I/O cycles, so byte and halfword I/O may fail on it (MACE specification + p. 167); iris does not model that. +- **Reset pins per device.** Which device each of the Bridge's four reset + pins serves is inferred (see above); no board document gives it. +- **Octane pin rule** (A and C to the device's pin, B and D to device ^ 4) is + inferred from the system's software, not established by a board document. +- **Bridge DMA** keeps its `st` lock and the `Weak` upgrade per call; that is a + separate project. +- **A held reset pin.** While a Bridge reset pin is held low the functions + still answer configuration cycles; iris acts on the assertion edge only. +- **AIC-7880 target state** (block size, write cache) is put back to the + post-bus-reset state on a load rather than saved (see restore above). +- **Resettable on PCI functions.** Ioc3, Isp1040, Rad1 and VirtioPci have + empty `Resettable` impls only because the device registry requires one. +- **A BAR that straddles a bridge window's edge** is not forwarded; what + hardware does is undefined. diff --git a/clanker/plans/pci-dma-port.md b/clanker/plans/pci-dma-port.md index 7d6cecb1f..da686c974 100644 --- a/clanker/plans/pci-dma-port.md +++ b/clanker/plans/pci-dma-port.md @@ -1,10 +1,13 @@ # PCI bus masters: one DMA trait, shared on the O2, per device on the Octane Status: plan, 2026-10-26. Step 1 done (the trait takes 64-bit bus addresses -and returns a status; the O2 is its first user). Steps 2-6 open. +and returns a status; the O2 is its first user). Steps 2-6 open. Since then +the Octane's Bridge and MACE both sit on one shared PCI layer that keeps +each function's interrupt line and DMA in its state: see `docs/pci.md` and +`clanker/plans/pci.md`, which overtake the steps below where they differ. Scope: how a PCI function reaches host memory and raises interrupts -(`src/pci.rs`: `PciBusDevice`, `PciDma`), the O2's MACE bridge +(`src/pci/function.rs`: `PciFunction`, `PciDma`), the O2's MACE bridge (`src/dev/mace_pci.rs`) and the Octane's Bridge/Xbridge to come. Sources: `docs/datasheets/octane/SGI-Bridge_ASIC_Specification-*.pdf` (chapters 3.3, 3.8, 3.9, 4, 11, appendix A) and `SGI-Xbridge_ASIC_Specification-*.pdf` (1.5, 7.3, diff --git a/clanker/plans/pci.md b/clanker/plans/pci.md new file mode 100644 index 000000000..e12eedf90 --- /dev/null +++ b/clanker/plans/pci.md @@ -0,0 +1,77 @@ +# PCI: what the shared layer delivers and what comes next + +Status: layer in place for the O2 and the Octane's onboard Bridge, 2026-11. +Steps (a) and (b) ride with the Fuel and Tezro branches, (c) is a follow-up, (d) +waits for the machine rework. The description of the layer is `docs/pci.md`; +its invariants are in `clanker/devices/pci-layer-design.md`; what is still +approximate is in `clanker/devices/pci-layer-open.md`. + +## What the layer delivers + +- One `PciBus` (`src/pci/`) for MACE and the Bridge: the function table, BAR + decode, interrupt combining, reset and snapshot state. A host decides only + how a CPU access becomes configuration cycles and a `ByteOrder` + (`src/pci/host.rs`), where an output goes, and what a function's DMA reaches. +- A PCI-to-PCI bridge function with a type 1 header. The Octane Bridge sends + type 1 cycles through its `0x028000` aperture (Bridge ASIC specification + rev 4.0, tables 26 and 43), and its 2 GB window carries the 32-bit alias of + PCI memory, so a BAR behind a bridge is reachable. +- RST# to every function on the Octane's cold reset, and to one device when + Bridge Control drops that device's reset pin. +- The `pci`, `bridge` and `heart` log modules and the Octane `pci` monitor + command. + +No machine fits a PCI-to-PCI bridge yet; the tests build one. + +## Steps + +### (a) Fuel moves its XBridge onto the layer + +The Fuel's XBridge (Xbridge ASIC specification rev 0.5, in `docs/datasheets/octane/`) +becomes a host of `PciBus` the way the Octane's Bridge is: its own config +cycles, byte order and interrupt pins, and the layer's function table and +decode. This belongs in the Fuel branch, which already owns the XBridge model. + +### (b) Tezro moves its PIC onto the layer + +The same for the Tezro's PIC (the Origin 3000 family's Crosstalk-to-PCI +chip). Two things to settle in that branch: the PIC's specification says how many +buses it has (if two, one host holds two `PciBus` values), and how its byte +order is chosen per window. + +### (c) A follow-up for real slots + +One change after (a) and (b), in this order: + +1. A generic XIO-to-PCI card: the XIO widget that carries a PCI bus behind it + (the "shoehorn" carrier), built as a `PciBus` host of its own. +2. The Octane's PCI card cage on top of it, with slots a guest can find by + scanning, through a `PciToPciBridge` where the hardware has one. +3. An NVMe controller placed in a card-cage slot. The onboard Bridge keeps only + what is built into the machine (the SCSI controllers, IOC3, RAD, and + virtio-serial when enabled); an add-in card never sits on it. + +Open questions for this step: how IRIX numbers the buses behind the cage +(IRIX 6.5.30 already scans a bus behind a bridge through the type 1 aperture), +and which interrupt pins the cage's slots reach. Neither is in a document we +hold; do not infer them from the onboard wiring. + +### (d) After the machine rework + +- RST# on the O2's cold reset. MACE has a `PCIRst_N` output and CRIME drives + system reset (MACE specification pp. 188 and 194), but no document says the + board joins them, so it stays inferred. +- A snapshot restore that re-creates devices rather than resetting them, and + the AIC-7880's three resets told apart: RST#, the chip's `CHIPRST`, and the + SCSI bus reset. Today a load calls the bus reset to put targets back. + +## Known open questions + +- Which device number the Bridge names in a master time-out for a cycle nobody + claims, and what an access to the reserved 16 MB to 1 GB of its 2 GB window + does (Bridge specification p. 52). Both are bus errors today. +- Whether MACE rev 0's 4-byte-aligned I/O cycles break byte and halfword I/O + (MACE specification p. 167), and the rule for MACE's 64-bit PIO. +- Which of the Bridge's four reset pins serves which device, and whether the + PROM or IRIX ever drops one. +- Whether the Bridge's pins pulse on a core reset or a Crossbow link reset. diff --git a/debug.md b/debug.md index 78c860259..bb228d34b 100644 --- a/debug.md +++ b/debug.md @@ -368,6 +368,13 @@ widget's link, refused Crossbow accesses, and accesses to physical addresses nothing is mapped at, with the PC; a device's own refusal is not listed). The `phys` commands work on the Octane as well. +On the Fuel the one controller is a two-channel ISP12160: `scsi` (or +`scsi0`) and `cow` act on channel 0, `scsi1` and `cow1` on channel 1, and +ids run 0 to 15 on each (`[scsi.17]` is id 1 on channel 1). `scsi notes` +shows the whole chip's command total and anything odd the driver did, and +has no `scsi1` form; `regs`, `wdt`, `defer`, `debug` and `dayna` are not +there. + Queue position `[0]` is the mounted disc, `[1]` the next on eject. A disc change reaches IRIX as a Unit Attention ("medium changed") on its next TEST UNIT READY; IRIX's `mediad` may still need a manual remount @@ -574,7 +581,7 @@ a forced termination; a normal stop saves them automatically. |---|---| | `phys mem [n]` / `mm` | Dump physical memory | | `phys dis [n]` / `md` | Disassemble physical memory | -| `phys map` | The decode table: pages by kind (untouched, filled but not yet mapped, shared, private), memory in use, slots faulted in, and the table as runs of devices. Every machine on `Physical` has the `phys` commands (Indy, Indigo2, O2, Octane) | +| `phys map` | The decode table: pages by kind (untouched, filled but not yet mapped, shared, private), memory in use, slots faulted in, and the table as runs of devices. Every machine on `Physical` has the `phys` commands (Indy, Indigo2, O2, Octane, Fuel) | | `phys trace ` / `trace ` | Log physical bus accesses | | `phys error ` | Stop on bus errors | | `phys hole ` | Stop on accesses to unmapped addresses | @@ -593,10 +600,18 @@ Per-module logging; all of it needs a developer build to produce output. | `log file ` | Send a module's log to a file | Modules: `net hpc3 seeq hal2 mc rex3 mips ioc scsi pdma vino dcb vc2 cmap -xmap bt445 scc ps2 rtc eeprom l1i l1d l2c bedrock`. Mask categories: `pdma` — +xmap bt445 scc ps2 rtc eeprom l1i l1d l2c bedrock pci bridge heart l1 usb`. `pci` logs every +configuration cycle on a PCI bus (MACE's, the Octane Bridge's) and +whether a function answered it. `bridge` logs the Octane Bridge's +register and translation-table accesses by name, and any transfer a PCI +function started that reached no memory, and each interrupt pin's edges +and the HEART vector it sets or clears. `heart` logs HEART's register +accesses, one counter read in 256 with the host's clock, and every +interrupt vector raised or lowered. Mask categories: `pdma` — `hal enet scsi`; `mips` — `insn tlb mem fpu cp0`; `l1i`/`l1d`/`l2c` — `hit miss op`; every module also takes `on`/`all`, `off`/`none` or a hex mask. -`bedrock` covers the Fuel's devices on its Bedrock hub. +`bedrock` covers the Fuel's devices on its Bedrock hub, `l1` its system +controller and `usb` its USB host controller. `IRIS_DEBUG_LOG` or `[debug] debug_log` set the same thing at startup. ### Performance and locks @@ -632,7 +647,7 @@ its SCSI controllers answer to `scsi`/`cow` and `scsi1`/`cow1` as described unde | `re dither [on\|off]` | Show or set whether the engine dithers, to compare renderings; on by default | | `gbe status` | GBE display engine: timing, frame buffer, tiles | | `isa status` | MACE ISA interrupts | -| `pci status` / `pci config ` | MACE PCI bridge / a function's configuration space | +| `pci status` / `pci config ` | The PCI bus (on the O2 with MACE's registers; the Octane has it too) / a function's configuration space | | `mec status` | MACE Ethernet: rings, receive buffers, frame counts | | `sdma status` | MACE serial DMA | | `maceps2 status` / `maceps2 traffic` | PS/2 ports / recent bytes each way | diff --git a/docs/fuel-ip34.md b/docs/fuel-ip34.md new file mode 100644 index 000000000..3622c313c --- /dev/null +++ b/docs/fuel-ip34.md @@ -0,0 +1,98 @@ +# SGI Fuel (IP34 board, IP35 software platform) + +Fuel is the IP34 board running the IP35 software platform. The emulated +configuration has one 500 MHz R14000 and 512 MiB of RAM in two fixed 256 MiB +banks. It includes Bedrock, two XBridge heads, an IOC3, and an ISP12160 with +two SCSI channels. The machine runs headless on its L1 or IOC3 serial console. + +## Configuration + +Copy [`fuel.toml.example`](../fuel.toml.example) to `fuel.toml`, provide the +PROM and disk files named there, then run `iris --config fuel.toml` from the +repository root. The sample selects `machine.profile = "fuel_ip34"` and +`machine.cpu = "r14000"`. The machine always runs its one R14000 (`r14000_500`, +the 500 MHz part); `cpu` may be left out, and any other value is refused. The sample +sets `banks = [256, 256]`, `headless = true`, `no_audio = true`, serial +ports 18880 and 18881, and `graphics.board = "none"`. The `[fuel]` settings +select the flash backing file, console, and optional L1 port and log. The +console options are `l1`, `ioc3a` and `ioc3b`. The Fuel's IOC3 has no +keyboard or mouse ports, so the console is always serial. + +The L1 answers firmware revision requests with 1.48.1 and brick serial +requests with the synthetic `EMUN00`; both are model choices, not values read +from a Fuel. `[fuel] l1_revision = "major.minor.patch"` and +`[fuel] brick_serial = "..."` replace them. Its system serial request returns +the machine's Ethernet address (`[network] mac`) as twelve uppercase +hexadecimal digits, the text the PROM converts back into that address. +IRIX's `numastatd` reads the revision at startup; without one it logs +`SNDRV_GET_ELSC_VERSION: I/O error`. + +The PROM decoder accepts a raw payload or a supported version-six, +single-segment stored or gzip container. The sample attaches a raw IRIX disk +at SCSI ID 1 and writes guest changes to its overlay. `[network] mode` defaults +to `nat`. `off` disconnects the link. `pcap` uses the shared `net::Link` backend +with a PCAP-enabled build and host capture support. The example sets `off`. +Set it to `nat` for the built-in gateway. + +## Observed boot and limits + +With the normal interpreter configuration, PROM 6.210 reports +`ALL DIAGS PASSED` and IRIX 6.5.30 has reached the root shell. The guest +reports a 500 MHz R14000 with 512 MiB, detects `ef0`, and passes the observed +NAT networking checks. During initial boot, IRIX's file indexer can keep the +guest busy while the serial console shows little activity. + +The PROM uses the R10000-family primary data cache as temporary memory before +it configures DRAM: it tags lines for the 32 KiB window `0x100000`-`0x107fff` +with Index Store Tag and installs them with Index Store Data, so its stack has +no memory behind it. The CPU model keeps those staged lines and serves cached +loads and stores that hit them, ahead of memory, until the PROM invalidates +them or the machine powers on; the memory controller has no special case for +the window. Without them the PROM takes a data bus error on its first stack +access. The O2's PROM does the same for its POST stack. The model's +approximations are listed in the cache-view note, +[`clanker/cpu/cache-lines-live-in-the-cpus-view.md`](../clanker/cpu/cache-lines-live-in-the-cpus-view.md). + +The ordinary interpreter configuration is the one with recorded Fuel boot +evidence. `lightning` and experimental `jitv2` builds are available, but no +Fuel guest boot with either is claimed here. + +## Graphics + +`[graphics] board = "v12"` (or `"v10"`) fits a VPro board in the graphics +slot, crossbar widget 13: a Fuel IRIX inventory lists its board under +xtalk/13, and Fuels have been reported with both the V12 and the V10. It is +the Octane's VPro model. Its interrupt packets cross the XBridge to Bedrock +like the PCI heads', its DMA enters Bedrock's inbound path as widget 13, and +a link reset of port 13 resets it. The processor interfaces' graphics +flow-control registers keep what IRIX writes but count no stores, so the +credit-starvation interrupt never fires. + +With PROM 6.210 the graphics diagnostic finds the board and passes. An IRIX +6.5.30 disk whose kernel includes the VPro driver lists `Graphics board: V12` +in `hinv` and brings the X login screen up on it. A disk installed for a +machine without graphics has no VPro driver in its kernel and runs as before. + +## Input + +The IOC3 carries a PS/2 keyboard and mouse, fed from host input; they are the +default, as in the Fuel's manual. `[fuel] usb_input = true` also plugs a USB +keyboard and mouse into the OHCI controller, and `ps2_input = false` leaves the +PS/2 pair out. Machine snapshots are unsupported. + +## Source basis + +The literal `0x53474972756c6573` is named `IIO_PROTECT_OVRRD_KEY` in public +Linux revision `db2ddb87143519e20a95aa36c60b36107b736a58`, +`arch/mips/include/asm/sn/sn0/hubio.h`. This establishes the literal's public +provenance, not the mapping of each Fuel register offset that uses it. Those +offsets remain approximate. The board composition port follows the project's +BSD-licensed QEMU SGI model in `hw/mips/sgi/sn1-node.c`. That source is a BSD-licensed implementation +reference, not proof of hardware behavior. + +The Hub-special head windows are compatibility approximations. Reads return +zero and writes are acknowledged without side effects. PROM 6.210 writes the +64-bit word `0x2000` at relative offsets `0x0` and `0x40` before discovering +DRAM; their hardware meaning remains unknown. Bedrock logging records +accesses. This keeps the existing model behavior explicit rather than +claiming a register implementation. diff --git a/docs/o2-ip32.md b/docs/o2-ip32.md index d61fb439a..eec01896f 100644 --- a/docs/o2-ip32.md +++ b/docs/o2-ip32.md @@ -99,6 +99,8 @@ O2's Ethernet is built in. The `--scsi1`, `--cdrom4` and related command-line options apply as well. An image that cannot be opened stops iris before the PROM runs, as on the other machines. +The PCI bus, its wiring and reset are described in [docs/pci.md](pci.md). + The controllers themselves are Adaptec AIC-7880s. [docs/dev/aic7880.md](dev/aic7880.md) describes the chip: how it works, its PCI configuration space, every register, the sequencer and the programs the PROM, IRIX and the BSDs download diff --git a/docs/octane-ip30-address-map.md b/docs/octane-ip30-address-map.md index b38d426a1..b624b0be6 100644 --- a/docs/octane-ip30-address-map.md +++ b/docs/octane-ip30-address-map.md @@ -59,6 +59,9 @@ The three window kinds are three views of the same XIO address space (§8): a short window shows the first 16 MB of a widget, a large window its first 2 GB, a huge window its first 64 GB. Widget 0 (the Xbow) has no huge window; its place is taken by memory and the large windows. +The code names them differently: `src/ip30.rs` calls the 2 GB window the +long window (`widget_long_base`) and the 64 GB one the large window +(`widget_large_base`). Memory starting at 512 MB is what lets the widgets' short windows, HEART and the PROM all sit below it, with the PROM at the address the CPU expects @@ -379,8 +382,8 @@ The Bridge decodes its XIO space as: | 0x8000_0000 – 0xBFFF_FFFF | PCI memory space, 64-bit-attribute variant of the same 1 GB | | 0x1_0000_0000 – 0x1_FFFF_FFFF | PCI I/O space, 4 GB | -So PCI memory address a is at physical 0xF_4000_0000 + a through widget -15's large window (0xF_0000_0000 + 0x4000_0000), and PCI I/O at +So PCI memory address a is at physical 0xF_C000_0000 + a through widget +15's large window (0xF_8000_0000 + 0x4000_0000), and PCI I/O at 0xF0_0000_0000 + 0x1_0000_0000 + a through its huge window. The control register's swap bits (§7) say whether PIO through those spaces is byte-swapped. The graphics widget's registers are at the start of its diff --git a/docs/octane-ip30.md b/docs/octane-ip30.md index ab9a253b1..373ca900c 100644 --- a/docs/octane-ip30.md +++ b/docs/octane-ip30.md @@ -5,7 +5,8 @@ controller and interrupt hub), a Crossbow switch joining the widgets, the onboard Bridge (XIO to PCI) carrying two QLogic ISP1040 SCSI controllers, the IOC3 I/O chip (serial ports, keyboard and mouse, the clock, Ethernet) and RAD audio, and a graphics board on one of the XIO ports. IRIS emulates a -single-processor Octane with a 195 MHz R10000. +single-processor Octane with a 195 MHz R10000. The Bridge's PCI bus, its +wiring and reset are described in [pci.md](pci.md). With the stock IP30 PROM (the 879,312-byte image, SHA-256 `ee4722ca6f9b821f…`) and an installed IRIX 6.5.30 disk, the machine boots to a serial login. diff --git a/docs/pci.md b/docs/pci.md new file mode 100644 index 000000000..d853db582 --- /dev/null +++ b/docs/pci.md @@ -0,0 +1,122 @@ +# PCI + +The O2 (through MACE) and the Octane (through the onboard Bridge) have PCI +buses. Both hosts own a `PciBus` (`src/pci/`), which holds the functions on +the bus, decodes their BARs, combines their interrupt pins and resets and +saves them. A host decides only what is its own: how a CPU access becomes +configuration cycles and byte order, where an interrupt output goes, and what +a function's DMA reaches. The bus has no byte order and knows no card by name. + +## Functions + +A function implements `PciFunction` (`src/pci/function.rs`): BAR reads and +writes, `reset` (RST#), and a `PciFunctionState` that holds its configuration +header and, once the bus attaches it, its interrupt line and DMA. The bus resets +a function's header; a function never resets its own, so the decode table cannot +go stale behind a BAR that moved. Headers change only through the bus: a +configuration write, a reset or a snapshot load, and each updates the decode +table and the BAR mappings. + +`PciBus::attach` gives the function its DMA and, if its header names an +interrupt pin, the line that pin drives, then calls `attached` for a function +that builds on them. A device number the wiring lacks, an +address already taken, or a pin the wiring does not route is a bug in the +machine and panics at attach. + +## Byte order + +A little-endian function behind a big-endian CPU needs a rule for how bytes +move, and the two hosts differ. A host chooses a `ByteOrder` value per window +(`src/pci/host.rs`): `Stream`, or `Values { width }` for a bus 4 or 8 bytes +wide. MACE is `Values { width: 4 }`: a byte access at CPU offset `a` reaches the +function at `a ^ 3`, a halfword at `a ^ 2`, a word as given (O2 Device Driver +Programming Guide 007-3443-002, pp. 397-398, Table 15-3). That guide does not +cover 64-bit accesses; iris sends one as two word cycles, the high word first, +and the rule is unknown. The Bridge takes its order from the window's Device(x) +register: the programmed swap bit gives `Stream`, and its 64-bit device bit +gives the width. A `BarDevice` the physical bus maps over a host window holds +its `ByteOrder` by value, so a BAR access is one match and the function's own +call. + +## Interrupts + +Every device pin and every board IRQ is a line routed to a host output, and +the bus tells the host the OR of what is routed to each output. A pin that +falls and rises before the host has seen the fall still reaches an edge host +as a 0 and then a 1: the Bridge sends its interrupt packet to HEART only on a +rising edge, and the guest may have cleared the bit in between. A rise that +falls again unseen is coalesced. No lock of ours is held while the host is +driven, and deliveries to one output are serialised and in order. + +## Reset + +`PciBus::reset` is RST#: each function's header, then `PciFunction::reset`, in +(device, function) order. Pin levels are left alone, so a function that is not +reset keeps its line. A restore is not a reset: it loads over what is there and +ends by telling the host the levels again. + +| Machine | RST# | Grade | +|---|---|---| +| O2 | Not yet: the O2's cold reset does not reach its PCI functions; that comes with a later change. MACE has no software PCI reset: the ISA ring-base bit resets the ISA block alone (MACE specification p. 123; PCI registers pp. 162-165). | When it comes, inferred. MACE has a `PCIRst_N` output (MACE specification p. 188) and a `SysReset` input (p. 194), and CRIME drives system and cold reset (CRIME specification, Table 8-2); that CRIME's reset reaches `SysReset` and that `PCIRst_N` follows it on the board is not documented. | +| Octane | On a cold reset, to every function. And when a write takes Bridge Control bit `24 + n` (`RST_PIN_N(3:0)`) from 1 to 0: RST# to the functions at PCI device `n` alone, with a logged warning. | The bits, their reset value (all high) and that a 1 drives the pin high are established (Bridge ASIC specification rev 4.0, Control register and pin list: four `BUS_RSTN` outputs, "PCI Bus reset signals"), so a 0 asserts RST#. Which device a pin serves is not documented and is inferred from the driver guide's `pciio_reset()` (007-3443-002, p. 550), which resets one card "without affecting other devices on the PCI bus". | +| Octane | Not on a Bridge core reset or a Crossbow link reset. | Not established. The Bridge specification says the core reset comes from the link reset (rev 4.0, section 7.5) and does not say whether the pins pulse. | + +## Wiring + +**O2**, MACE bus 0. Outputs are CRIME inputs (MACE specification Tables 85, 87 +and 90; established as software, and the board's INTB-D wiring is not known). + +| Device | INTA | INTB | INTC | INTD | Use | +|---|---|---|---|---|---| +| 1 | CRIME 8 | | | | Internal SCSI controller 0 | +| 2 | CRIME 9 | | | | Internal SCSI controller 1 | +| 3 | CRIME 10 | CRIME 13 | CRIME 14 | CRIME 15 | The PCI option slot | +| 4 | CRIME 11 | | | | Not populated on an O2; iris keeps it spare for virtio-serial | + +**Octane**, Bridge bus 0. Outputs are the Bridge's eight pins. Pins A and C of +device `d` reach pin `d`, B and D reach `d ^ 4`: inferred from software, not +established. `BoardIrq(i)` is pin `i`, shared with the GPIO inputs; the IOC3's +SuperIO interrupt is `BoardIrq(4)`. Devices 0 and 1 are the ISP1040s, 2 is the +IOC3, 3 is the RAD1, and 4 is virtio-serial when enabled. + +## Octane windows + +The CPU reaches PCI memory on the Bridge two ways (Bridge ASIC specification +rev 4.0; offsets in [octane-ip30-address-map.md](octane-ip30-address-map.md), section +8). A device window in the Bridge's own widget space, set by `Device(x)`; and +the 2 GB window HEART gives widget 15 at `0xF_8000_0000`, whose first 16 MB is +the widget's small window and whose range from offset `0x4000_0000` is the +32-bit alias of the low 1 GB of PCI memory. A BAR in either is mapped straight +to its function. The rest of the 2 GB window (16 MB to 1 GB) is reserved in the +specification (p. 52), which does not say what an access does; iris ends it in +a bus error, an approximation. The 64-bit alias and PCI I/O space, reached only +through the 64 GB window, are not modelled. The external ATE SSRAM (`0x080000`) +is not fitted on the Octane and reads as all ones, which is what the PROM +expects when it sizes it. + +## PCI-to-PCI bridges + +`PciToPciBridge` is a function with a type 1 header and a bus behind it. The +bus in front merges the BARs behind the bridge that lie wholly inside an enabled +window into its own decode table and runs; a BAR that straddles a window's edge +is not forwarded, because what a bridge does with one is not defined. A pin of +the function at device `d` behind the bridge reaches pin `(pin + d) mod 4` of +the bridge's device, and its DMA is the bridge's. Raising bridge control bit 6 +resets the bus behind it, and RST# on the bus in front reaches behind it too. +MACE's configuration cycles for a bus number other than 0 go to the bus firmware +has numbered that far. A configuration cycle that reaches the bus behind a +bridge and finds no function ends in master abort at the bridge: a read returns +all ones, a write is dropped, and the bridge's secondary status records received +master abort. On the Octane, the Bridge's type 1 aperture (0x028000 in its +widget space, Bridge specification table 43) sends a cycle to the bus and device +its PCI Type 1 Configuration register names (table 26); a bus number no bridge's +secondary to subordinate range holds, including 0, ends in a master time-out. +IRIX 6.5.30 scans a bus behind a bridge this way. The header is the type 1 +header only (the caller names the PCI identity); the upper halves of the +prefetchable and I/O windows read zero. No machine builds one yet. + +## Monitor + +`pci status` lists each function: address, identity, command, assigned BARs +and where its pin goes (on the Octane too, since both machines share the bus); +`pci config ` dumps a function's header. diff --git a/fuel.toml.example b/fuel.toml.example new file mode 100644 index 000000000..8ac609a09 --- /dev/null +++ b/fuel.toml.example @@ -0,0 +1,44 @@ +# SGI Fuel (IP34 board, IP35 software platform), serial-only setup. +# PROM 6.210 and IRIX 6.5.30 have been observed reaching the root shell. +# Copy to fuel.toml and run from the repository root: iris --config fuel.toml +# Create ignore/fuel/ and supply your decoded raw PROM payload and IRIX disk. +# PROM also accepts supported version-six single-segment stored/gzip containers. +# No PROM or disk images are included. Convert QCOW2 disks to raw first. + +prom = "ignore/fuel/ip35prom.payload.bin" +nvram = "ignore/fuel/nvram.bin" +banks = [256, 256] +headless = true +no_audio = true +monitor_port = 18888 +serial_port_a = 18880 +serial_port_b = 18881 + +[machine] +profile = "fuel_ip34" +# The Fuel's one processor, the R14000 at 500 MHz with a 2 MiB secondary +# cache (`r14000_500`); other values are refused. Unset takes the same. +cpu = "r14000" + +[graphics] +# "v12" or "v10" fits a VPro in the graphics slot; IRIX needs a kernel with +# the VPro driver. The console stays on serial either way. +board = "none" + +[fuel] +flash = "ignore/fuel/flash.bin" +console = "l1" +l1_port = 18882 + +[network] +# Disconnected cable. For NAT, use mode = "nat". +# Keep the installed disk's working ef0 address and default route. +# NAT can adopt its static gateway before routing begins. +# Initial defaults: guest 192.168.0.2/24, gateway 192.168.0.1. +mode = "off" + +[scsi.1] +path = "ignore/fuel/irix.raw" +cdrom = false +# Keep the base disk unchanged. Guest writes go to irix.raw.overlay. +overlay = true diff --git a/src/config.rs b/src/config.rs index ad30b6b0c..f770c5a92 100644 --- a/src/config.rs +++ b/src/config.rs @@ -628,6 +628,8 @@ pub enum MachineProfile { /// a test widget. Configured by the shared keys and `[origin2000]`; see /// `src/ip27_onyx2.rs`. Onyx2Ip27, + /// Serial-only Fuel IP34 board using the IP35 software platform. + FuelIp34, } impl MachineProfile { @@ -651,6 +653,7 @@ impl MachineProfile { Self::Origin200Ip29 | Self::Origin2000Ip27 | Self::Onyx2Ip27 => &[0, 64, 128, 256, 512], Self::O2Ip32 => &[0, 32, 64, 128, 256], Self::Indigo2Ip28 => MC_BANK_SIZES, + Self::FuelIp34 => &[0, 256], _ => &[0, 8, 16, 32, 64, 128], } } @@ -675,7 +678,7 @@ impl MachineProfile { /// Every profile, in the order `workstation::build` lists them. `ALL` is /// only the ones the GUI offers. - pub const EVERY: [Self; 8] = [ + pub const EVERY: [Self; 9] = [ Self::IndyIp24, Self::Indigo2Ip22, Self::Indigo2Ip28, @@ -684,6 +687,7 @@ impl MachineProfile { Self::Origin200Ip29, Self::Origin2000Ip27, Self::Onyx2Ip27, + Self::FuelIp34, ]; /// The name `[machine] profile` and `--machine` give this profile. @@ -697,6 +701,7 @@ impl MachineProfile { Self::Origin200Ip29 => "origin200_ip29", Self::Origin2000Ip27 => "origin2000_ip27", Self::Onyx2Ip27 => "onyx2_ip27", + Self::FuelIp34 => "fuel_ip34", } } @@ -754,6 +759,7 @@ impl MachineProfile { Self::Origin200Ip29 => "SGI Origin 200 (IP29)", Self::Origin2000Ip27 => "SGI Origin 2000 (IP27)", Self::Onyx2Ip27 => "SGI Onyx2 (IP27)", + Self::FuelIp34 => "SGI Fuel (IP34)", } } @@ -763,7 +769,7 @@ impl MachineProfile { /// connector), one on the Indy. pub fn scsi_controllers(self) -> u8 { match self { - Self::Indigo2Ip22 | Self::O2Ip32 | Self::OctaneIp30 | Self::Origin200Ip29 | Self::Origin2000Ip27 | Self::Onyx2Ip27 => 2, + Self::Indigo2Ip22 | Self::O2Ip32 | Self::OctaneIp30 | Self::Origin200Ip29 | Self::Origin2000Ip27 | Self::Onyx2Ip27 | Self::FuelIp34 => 2, _ => 1, } } @@ -772,18 +778,19 @@ impl MachineProfile { /// buses are wide, with ids 0 to 15 and the controller on 0. The Indy's /// and Indigo2's WD33C93A chips are narrow, with ids 0 to 7. The O2's /// Adaptec buses are wide too, but stay at 7 until they are widened on - /// purpose. + /// purpose. The Fuel's ISP12160 buses are wide. pub fn max_scsi_id(self) -> u8 { match self { - Self::OctaneIp30 | Self::Origin200Ip29 | Self::Origin2000Ip27 | Self::Onyx2Ip27 => 15, + Self::OctaneIp30 | Self::Origin200Ip29 | Self::Origin2000Ip27 | Self::Onyx2Ip27 | Self::FuelIp34 => 15, _ => 7, } } - /// How many memory banks the machine has: four on all of them (the MC's - /// four banks; the O2's four DIMM pairs). `banks` has room for more. + /// How many memory banks the machine has: four on the MC, HEART and O2 + /// (the O2's four DIMM pairs), two on Fuel. `banks` has room for more; + /// unused entries must be zero. pub fn memory_banks(self) -> usize { - 4 + if self == Self::FuelIp34 { 2 } else { 4 } } pub fn guinness(self) -> bool { @@ -798,6 +805,7 @@ impl MachineProfile { Self::Origin200Ip29 | Self::Origin2000Ip27 | Self::Onyx2Ip27 => crate::ip27_machine::CPUS, Self::O2Ip32 => crate::ip32_machine::CPUS, Self::Indigo2Ip28 => crate::machine::IP28_CPUS, + Self::FuelIp34 => crate::ip35_machine::CPUS, Self::IndyIp24 | Self::Indigo2Ip22 => crate::machine::CPUS, } } @@ -837,18 +845,18 @@ pub enum GraphicsBoard { /// VPro V8: BUZZ A, 128 MB. Octane only. #[clap(name = "v8", alias = "vpro:v8", alias = "vpro_v8")] V8, - /// VPro V10: BUZZ B, 32 MB. Octane only. It shipped only in the Octane2 - /// and needs Crossbow 1.4, so the machine takes the Octane2's Crossbow and - /// Bridge revisions. + /// VPro V10: BUZZ B, 32 MB. Octane and Fuel. On the Octane it shipped + /// only in the Octane2 and needs Crossbow 1.4, so the machine takes the + /// Octane2's Crossbow and Bridge revisions. #[clap(name = "v10", alias = "vpro:v10", alias = "vpro_v10")] V10, - /// VPro V12: BUZZ B, 128 MB. Octane only. It shipped only in the Octane2 - /// and needs Crossbow 1.4, so the machine takes the Octane2's Crossbow and - /// Bridge revisions. + /// VPro V12: BUZZ B, 128 MB. Octane and Fuel. On the Octane it shipped + /// only in the Octane2 and needs Crossbow 1.4, so the machine takes the + /// Octane2's Crossbow and Bridge revisions. #[clap(name = "v12", alias = "vpro:v12", alias = "vpro_v12")] V12, /// No graphics board: the machine runs on its serial console. Octane - /// only. + /// and Fuel. #[clap(name = "none")] None, } @@ -911,6 +919,7 @@ impl GraphicsBoard { match profile { MachineProfile::OctaneIp30 => self.is_impact() || self.is_vpro() || self == Self::None, MachineProfile::Origin200Ip29 | MachineProfile::Origin2000Ip27 | MachineProfile::Onyx2Ip27 => self == Self::None, + MachineProfile::FuelIp34 => matches!(self, Self::V10 | Self::V12 | Self::None), MachineProfile::Indigo2Ip22 => !self.is_vpro() && self != Self::None, _ => !self.is_vpro() && !matches!(self, Self::Extreme | Self::None), } @@ -921,6 +930,7 @@ impl GraphicsBoard { match profile { MachineProfile::OctaneIp30 => Self::SolidImpact, MachineProfile::Origin200Ip29 | MachineProfile::Origin2000Ip27 | MachineProfile::Onyx2Ip27 => Self::None, + MachineProfile::FuelIp34 => Self::None, _ => Self::Newport, } } @@ -980,7 +990,7 @@ impl std::fmt::Display for GraphicsBoard { #[serde(deny_unknown_fields)] pub struct GraphicsSection { /// Graphics board: Newport, GR2 XZ/Extreme, IMPACT Solid/High/Maximum, - /// or none (Octane only). Unset: the machine's default + /// or none (Octane and Fuel). Unset: the machine's default /// ([`GraphicsBoard::default_for`]); read it with /// [`MachineConfig::graphics_board`]. Extreme is restricted to IP22, /// the Octane to IMPACT or none; non-Newport boards require one head @@ -1748,6 +1758,80 @@ mod scsi_keys { } } +/// Fuel's independent serial connections. The selected console serves CI and serial_log. +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum FuelConsole { + #[default] + L1, + Ioc3A, + Ioc3B, +} + +#[derive(Debug, Clone, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct FuelSection { + /// Persistent 16 MiB boot flash store; absent means volatile flash. + #[serde(default)] + pub flash: Option, + #[serde(default)] + pub console: FuelConsole, + /// L1 host console TCP port; IOC3 keeps serial_port_a and serial_port_b. + #[serde(default)] + pub l1_port: Option, + /// Separate L1 log when an IOC3 port is selected as console. + #[serde(default)] + pub l1_log: Option, + /// Fit the PS/2 keyboard and mouse controller in the onboard IOC3 and + /// feed it the host's keyboard and mouse. + #[serde(default = "default_fuel_input")] + pub ps2_input: bool, + /// Plug a USB keyboard and mouse into the OHCI root ports and feed them + /// the same host input. Off by default: the Fuel's manual and the PROM + /// treat PS/2 as the standard keyboard and mouse. + #[serde(default)] + pub usb_input: bool, + /// L1 firmware revision as "major.minor.patch", reported to request + /// 0x0005. Unset: 1.48.1. + #[serde(default)] + pub l1_revision: Option, + /// Brick serial reported to L1 request 0x0004, at most 37 bytes. Unset: + /// the marked synthetic `EMUN00`. + #[serde(default)] + pub brick_serial: Option, +} + +fn default_fuel_input() -> bool { true } + +impl Default for FuelSection { + fn default() -> Self { + Self { + flash: None, + console: FuelConsole::default(), + l1_port: None, + l1_log: None, + ps2_input: true, + usb_input: false, + l1_revision: None, + brick_serial: None, + } + } +} + +impl FuelSection { + /// Left out of a written config when nothing in it was set. + fn is_default(&self) -> bool { + self.flash.is_none() + && self.console == FuelConsole::default() + && self.l1_port.is_none() + && self.l1_log.is_none() + && self.ps2_input + && !self.usb_input + && self.l1_revision.is_none() + && self.brick_serial.is_none() + } +} + /// Top-level machine configuration. /// /// Field order matters for TOML export: the `toml` serializer requires every @@ -2002,6 +2086,10 @@ pub struct MachineConfig { #[serde(default, skip_serializing_if = "Origin2000Section::is_default")] pub origin2000: Origin2000Section, + /// Serial console and flash storage for the Fuel. + #[serde(default, skip_serializing_if = "FuelSection::is_default")] + pub fuel: FuelSection, + /// Host performance tuning (`[perf]` section). #[serde(default)] pub perf: PerfConfig, @@ -2232,6 +2320,7 @@ impl Default for MachineConfig { octane: OctaneSection::default(), origin200: Origin200Section::default(), origin2000: Origin2000Section::default(), + fuel: FuelSection::default(), perf: PerfConfig::default(), clock: ClockConfig::default(), rtc_offset: RtcOffset::default(), @@ -2284,6 +2373,9 @@ impl MachineConfig { MachineProfile::Origin200Ip29 => Err("[virtio_serial] is not wired on the Origin 200".into()), MachineProfile::Origin2000Ip27 => Err("[virtio_serial] is not wired on the Origin 2000".into()), MachineProfile::Onyx2Ip27 => Err("[virtio_serial] is not wired on the Onyx2".into()), + MachineProfile::FuelIp34 => Err( + "[virtio_serial] is not wired on the Fuel: its XBridge has no virtio-pci attachment".into(), + ), MachineProfile::O2Ip32 => Err( "[virtio_serial] is not wired on the O2 yet: the virtio-pci transport exists, but the \ MACE PCI bridge has no interrupt route for a third slot (see docs/dev/virtio-serial.md)" @@ -2433,7 +2525,7 @@ impl MachineConfig { /// Validate bank sizes, returns a description of any errors. /// The graphics board: `[graphics] board`, or the machine's default - /// when it is not set (Solid IMPACT on the Octane, Newport elsewhere). + /// (Solid IMPACT on Octane, none on Fuel, Newport elsewhere). pub fn graphics_board(&self) -> GraphicsBoard { self.graphics.board.unwrap_or_else(|| GraphicsBoard::default_for(self.machine.profile)) } @@ -2487,6 +2579,8 @@ impl MachineConfig { format!("graphics.board \"{name}\" is not modelled on the Origin 2000 (valid: none)") } else if self.machine.profile == MachineProfile::Onyx2Ip27 { format!("graphics.board \"{name}\" is not modelled on the Onyx2 (valid: none)") + } else if self.machine.profile == MachineProfile::FuelIp34 { + format!("graphics.board \"{name}\" is not a Fuel board (valid: v12, v10, none)") } else { format!("graphics.board \"{name}\" is only valid on the Octane") }); @@ -2528,9 +2622,9 @@ impl MachineConfig { "scsi.{} and scsi.{} both name {}", other, key, scsi_label(controller, id) )); } - if dev.is_daynaport() && matches!(self.machine.profile, MachineProfile::O2Ip32 | MachineProfile::Origin200Ip29 | MachineProfile::Origin2000Ip27 | MachineProfile::Onyx2Ip27) { + if dev.is_daynaport() && matches!(self.machine.profile, MachineProfile::O2Ip32 | MachineProfile::Origin200Ip29 | MachineProfile::Origin2000Ip27 | MachineProfile::Onyx2Ip27 | MachineProfile::FuelIp34) { return Err(format!( - "scsi.{}: kind = \"daynaport\" is not available on the O2 (its Ethernet is built in)", key + "scsi.{}: kind = \"daynaport\" is not available on {} (its Ethernet is built in)", key, self.machine.profile.label() )); } if dev.is_daynaport() && self.network.mode == NetMode::Stream { @@ -2592,6 +2686,19 @@ impl MachineConfig { return Err(format!("bank{i} size 256 MB: the PROM disables 256 MB banks on the 180 MHz R10000 module (use 128 and 128, or the r12000 module)")); } } + } else if self.machine.profile == MachineProfile::FuelIp34 { + if self.banks[..2] != [256, 256] { + return Err("Fuel currently requires banks = [256, 256] (512 MiB)".into()); + } + if !board.supports(MachineProfile::FuelIp34) { + return Err(format!("graphics.board \"{}\" is not a Fuel board (valid: v12, v10, none)", board.as_str())); + } + if let Some(text) = &self.fuel.l1_revision { + crate::dev::l1::parse_revision(text).map_err(|e| format!("fuel.l1_revision: {e}"))?; + } + if self.fuel.brick_serial.as_ref().is_some_and(|s| s.len() > 37 || s.contains('\0')) { + return Err("fuel.brick_serial must be at most 37 bytes with no NUL".into()); + } } else if self.machine.profile == MachineProfile::OctaneIp30 { // A HEART bank is a matched DIMM pair: 64 MB to 2 GB, filled from // bank 0 without a gap, 8 GB at most (SGI's memory upgrade guide). @@ -2637,8 +2744,8 @@ impl MachineConfig { } } match (self.network.mode, self.machine.profile) { - (NetMode::Off, p) if p != MachineProfile::O2Ip32 && p != MachineProfile::OctaneIp30 && p != MachineProfile::Origin200Ip29 && p != MachineProfile::Origin2000Ip27 && p != MachineProfile::Onyx2Ip27 => { - return Err("network.mode = \"off\" is only available on the O2, the Octane and the Origin 200 and 2000 for now; the Indy and Indigo2 always have their Ethernet".into()); + (NetMode::Off, p) if p != MachineProfile::O2Ip32 && p != MachineProfile::OctaneIp30 && p != MachineProfile::Origin200Ip29 && p != MachineProfile::Origin2000Ip27 && p != MachineProfile::Onyx2Ip27 && p != MachineProfile::FuelIp34 => { + return Err("network.mode = \"off\" is only available on the O2, the Octane, the Fuel and the Origin 200 and 2000 for now; the Indy and Indigo2 always have their Ethernet".into()); } (NetMode::Stream, _) => { let ep = self.network.stream.as_deref() @@ -3702,7 +3809,7 @@ mod export_tests { assert_eq!(apply(&["--ip22"]).machine.profile, MachineProfile::Indigo2Ip22); assert!(Cli::try_parse_from(["iris", "--ip22", "--machine", "o2_ip32"]).is_err()); let e = MachineProfile::from_name("vax").unwrap_err(); - assert_eq!(e, "unknown machine \"vax\" (valid: indy_ip24, indigo2_ip22, indigo2_ip28, o2_ip32, octane_ip30, origin200_ip29, origin2000_ip27, onyx2_ip27)"); + assert_eq!(e, "unknown machine \"vax\" (valid: indy_ip24, indigo2_ip22, indigo2_ip28, o2_ip32, octane_ip30, origin200_ip29, origin2000_ip27, onyx2_ip27, fuel_ip34)"); let list = MachineProfile::list(); assert_eq!(list.lines().count(), MachineProfile::EVERY.len()); assert!(list.starts_with("indy_ip24 SGI Indy (IP24)\n"), "{list}"); @@ -3711,7 +3818,7 @@ mod export_tests { for p in MachineProfile::EVERY { match p { MachineProfile::IndyIp24 | MachineProfile::Indigo2Ip22 | MachineProfile::Indigo2Ip28 - | MachineProfile::O2Ip32 | MachineProfile::OctaneIp30 | MachineProfile::Origin200Ip29 | MachineProfile::Origin2000Ip27 | MachineProfile::Onyx2Ip27 => {} + | MachineProfile::O2Ip32 | MachineProfile::OctaneIp30 | MachineProfile::Origin200Ip29 | MachineProfile::Origin2000Ip27 | MachineProfile::Onyx2Ip27 | MachineProfile::FuelIp34 => {} } let cfg: MachineConfig = toml::from_str(&format!("[machine]\nprofile = \"{}\"", p.name())).unwrap(); assert_eq!(cfg.machine.profile, p); @@ -3726,6 +3833,7 @@ mod export_tests { let accepts = |p: MachineProfile, cpu: &str| p.part(Some(cpu)).is_ok(); assert!(accepts(MachineProfile::OctaneIp30, "r14000") && !accepts(MachineProfile::OctaneIp30, "r5000")); assert!(accepts(MachineProfile::IndyIp24, "r5000") && !accepts(MachineProfile::IndyIp24, "r10000")); + assert!(accepts(MachineProfile::FuelIp34, "r14000") && !accepts(MachineProfile::FuelIp34, "r10000")); // The counts are what the checks in validate() and the Octane // constructor allowed before they moved here: one on every machine @@ -4158,3 +4266,131 @@ mod ci_overlay_tests { assert!(a.contains(&format!("iris-ci-{}-", std::process::id())), "{a}"); } } + +#[cfg(test)] +mod fuel_input_tests { + use super::*; + + /// A config saved for another machine must not grow a `[fuel]` table, + /// and a Fuel's own settings must survive a save and reload. + #[test] + fn only_a_configured_fuel_section_is_saved_and_it_round_trips() { + let other = MachineConfig::default(); + let saved = toml::to_string(&other).unwrap(); + assert!(!saved.contains("[fuel]"), "{saved}"); + let mut fuel = MachineConfig::default(); + fuel.fuel.console = FuelConsole::Ioc3B; + fuel.fuel.flash = Some("flash.bin".into()); + fuel.fuel.l1_port = Some(18882); + fuel.fuel.l1_log = Some("l1.log".into()); + fuel.fuel.usb_input = true; + fuel.fuel.ps2_input = false; + let saved = toml::to_string(&fuel).unwrap(); + assert!(saved.contains("[fuel]"), "{saved}"); + let loaded: MachineConfig = toml::from_str(&saved).unwrap(); + assert_eq!(loaded.fuel.console, FuelConsole::Ioc3B); + assert_eq!(loaded.fuel.flash.as_deref(), Some("flash.bin")); + assert_eq!(loaded.fuel.l1_port, Some(18882)); + assert_eq!(loaded.fuel.l1_log.as_deref(), Some("l1.log")); + assert!(!loaded.fuel.ps2_input && loaded.fuel.usb_input); + assert!(!MachineProfile::ALL.contains(&MachineProfile::FuelIp34)); + } + + /// PS/2 is the Fuel's standard keyboard and mouse; USB is opt-in, and a + /// `[fuel]` table that names neither key keeps both defaults. + #[test] + fn fuel_input_defaults_to_ps2_with_usb_opt_in() { + let none: MachineConfig = toml::from_str("[fuel]\nflash = \"flash.bin\"\n").unwrap(); + assert!(none.fuel.ps2_input && !none.fuel.usb_input); + assert!(MachineConfig::default().fuel.is_default()); + let usb: MachineConfig = toml::from_str("[fuel]\nusb_input = true\n").unwrap(); + assert!(usb.fuel.ps2_input && usb.fuel.usb_input); + assert!(!usb.fuel.is_default(), "an opt-in is written back"); + } + + /// The Fuel's processor is fixed, so `machine.cpu` may only name it (or the + /// family, as the key did before the R14000 had a name); the default + /// `r4400` is refused rather than silently ignored. + /// The L1 identity keys parse, validate and round trip; unset they are + /// left out of a written config. + #[test] + fn fuel_l1_identity_keys_are_validated_and_round_trip() { + let base = "banks = [256, 256]\n[machine]\nprofile = \"fuel_ip34\"\ncpu = \"r14000\"\n"; + let ok: MachineConfig = toml::from_str(&format!( + "{base}[fuel]\nl1_revision = \"1.50.2\"\nbrick_serial = \"EMUN03\"\n" + )).unwrap(); + ok.validate().unwrap(); + let saved = toml::to_string(&ok).unwrap(); + let back: MachineConfig = toml::from_str(&saved).unwrap(); + assert_eq!(back.fuel.l1_revision.as_deref(), Some("1.50.2")); + assert_eq!(back.fuel.brick_serial.as_deref(), Some("EMUN03")); + let plain: MachineConfig = toml::from_str(base).unwrap(); + plain.validate().unwrap(); + assert!(plain.fuel.l1_revision.is_none() && plain.fuel.brick_serial.is_none()); + let bad_rev: MachineConfig = toml::from_str(&format!("{base}[fuel]\nl1_revision = \"1.48\"\n")).unwrap(); + assert!(bad_rev.validate().unwrap_err().contains("fuel.l1_revision")); + let long = "x".repeat(38); + let bad_serial: MachineConfig = toml::from_str(&format!("{base}[fuel]\nbrick_serial = \"{long}\"\n")).unwrap(); + assert!(bad_serial.validate().unwrap_err().contains("fuel.brick_serial")); + } + + #[test] + fn fuel_takes_only_its_r14000() { + let mut cfg = MachineConfig::default(); + cfg.machine.profile = MachineProfile::FuelIp34; + cfg.banks.fill(0); + cfg.banks[0] = 256; + cfg.banks[1] = 256; + cfg.scsi.clear(); + // Leaving `cpu` out takes the Fuel's own part. + cfg.machine.cpu = None; + cfg.validate().unwrap(); + assert_eq!(cfg.cpu(), &crate::cpu::parts::R14000_500); + for name in ["r14000", "r14000_500"] { + cfg.machine.cpu = Some(name.into()); + cfg.validate().unwrap(); + assert_eq!(cfg.cpu(), &crate::cpu::parts::R14000_500, "{name}"); + } + for name in ["r10000", "r12000_360", "r14000_600"] { + cfg.machine.cpu = Some(name.into()); + assert!(cfg.validate().unwrap_err().contains("r14000_500"), "{name}"); + } + } + + #[test] + fn fuel_requires_its_fixed_memory_population_and_a_vpro_or_no_graphics() { + let mut cfg = MachineConfig::default(); + cfg.machine.profile = MachineProfile::FuelIp34; + cfg.banks.fill(0); + cfg.banks[0] = 256; + cfg.banks[1] = 256; + cfg.scsi.clear(); + assert_eq!(cfg.graphics_board(), GraphicsBoard::None); + cfg.network.mode = NetMode::Off; + cfg.validate().unwrap(); + cfg.network.mode = NetMode::Pcap; + cfg.validate().unwrap(); + cfg.graphics.board = Some(GraphicsBoard::Newport); + assert!(cfg.validate().unwrap_err().contains("none")); + for board in [GraphicsBoard::V12, GraphicsBoard::V10] { + cfg.graphics.board = Some(board); + cfg.validate().unwrap(); + } + cfg.graphics.board = Some(GraphicsBoard::V6); + assert!(cfg.validate().unwrap_err().contains("not a Fuel board")); + cfg.graphics.board = None; + cfg.banks[1] = 0; + assert!(cfg.validate().unwrap_err().contains("256, 256")); + } + + #[test] + fn fuel_accepts_wide_scsi_ids_on_both_channels_without_changing_narrow_profiles() { + let text = "banks = [256, 256]\n[machine]\nprofile = \"fuel_ip34\"\ncpu = \"r14000\"\n[scsi.15]\npath = \"disk.raw\"\n[scsi.31]\npath = \"disk1.raw\"\n"; + let mut cfg: MachineConfig = toml::from_str(text).unwrap(); + cfg.validate().unwrap(); + // The Octane's buses are wide as well; the Indy's WD33C93A is narrow. + cfg.machine.profile = MachineProfile::IndyIp24; + cfg.machine.cpu = Some("r4400".into()); + assert!(cfg.validate().unwrap_err().contains("out of range")); + } +} diff --git a/src/cpu/jitv2/comp.rs b/src/cpu/jitv2/comp.rs index d5d009ec8..595750f31 100644 --- a/src/cpu/jitv2/comp.rs +++ b/src/cpu/jitv2/comp.rs @@ -116,6 +116,8 @@ pub(crate) enum Snapshot { Taken(u64), /// A word could not be read (not RAM right now). Unreadable, + /// A CPU's cache holds lines in the page, under this generation. + Held(u64), /// The generation moved under every attempt. Busy, } @@ -129,6 +131,10 @@ pub(crate) fn snapshot_page(page: &PhysicalCodePage, bus: &dyn BusDevice, phys_b const MAX_SEQLOCK_RETRIES: u32 = 8; for _ in 0..MAX_SEQLOCK_RETRIES { let gen0 = page.current_gen(); + // A CPU's cache holds lines in the page (`GEN_HELD`). + if gen0 >= crate::cpu::jitv2::jitv2::GEN_HELD { + return Snapshot::Held(gen0); + } // A seqlock reader: the words are read after `gen0` and before // `gen1` (acquire), and every writer bumps the generation after its // bytes, with release. Then a generation that held means these are @@ -199,11 +205,27 @@ fn prepare_multi_entry_compile( // §13.3 step 2: seqlock-snapshot the page bytes. let mut words = [0u32; ENTRIES_PER_PAGE]; let gen_snap = match snapshot_page(page, &**bus, phys_base, &mut words) { - Snapshot::Taken(gen) => gen, + Snapshot::Taken(gen) => { + // The lines are gone: lift the held publish's denials (below). + if page.entry_gen() >= crate::cpu::jitv2::jitv2::GEN_HELD { + page.allow_all(); + } + gen + } Snapshot::Unreadable => { page.mark_prepare_bounced(); return PrepareOutcome::Done(false); // page not readable right now; retry later } + Snapshot::Held(gen) => { + // Publish no code for this generation and deny every offset, so + // arrivals interpret the page without asking again until the + // release moves the generation (`step_jit`'s denylist gate). + if page.publish(&[0; crate::cpu::jitv2::BITMAP_WORDS], std::ptr::null(), gen, 0, 0, req.compiled_for_fr1) { + page.denylist_all(); + } + page.mark_prepare_bounced(); + return PrepareOutcome::Done(false); + } Snapshot::Busy => { // A sufficiently hot self-modifying/shared-DMA page can bounce here // EVERY single attempt (gen keeps changing faster than @@ -976,6 +998,26 @@ mod tests { assert!(!page.is_denylisted(4)); } + /// A page whose counter carries `GEN_HELD` compiles nothing and denies + /// every offset, and compiles once the mark comes off. + #[test] + fn a_held_page_compiles_once_its_lines_go() { + let _stable = crate::cpu::jitv2::opcode_support::table_stable(); + let bus: Arc = Arc::new(AddiuDevice); + let counter = AtomicU64::new(crate::cpu::jitv2::jitv2::GEN_HELD); + let mut page = PhysicalCodePage::new(0, &counter as *const AtomicU64); + let mut compile = |page: &mut PhysicalCodePage| { + page.mark_requested(4); + let req = CompileRequest { page: page as *mut PhysicalCodePage, compiled_for_fr1: true }; + handle_request_for_test(&req, &bus, &mut Analyzer::new(), &mut Codegen::new()); + }; + compile(&mut page); + assert!(!page.is_runnable(4) && page.is_denylisted(4), "held: interpreted, not asked again"); + counter.fetch_add(1u64.wrapping_sub(crate::cpu::jitv2::jitv2::GEN_HELD), std::sync::atomic::Ordering::Release); + compile(&mut page); + assert!(page.is_runnable(4) && !page.is_denylisted(4), "released: compiled"); + } + #[test] fn a_second_incremental_compile_does_not_drop_the_first_entry() { // §13's one-function-per-page model fully replaces `func` on every diff --git a/src/cpu/jitv2/jitv2.rs b/src/cpu/jitv2/jitv2.rs index 6442262fa..01455afab 100644 --- a/src/cpu/jitv2/jitv2.rs +++ b/src/cpu/jitv2/jitv2.rs @@ -228,6 +228,13 @@ pub fn push_hot_request(hot: &crossbeam_queue::ArrayQueue, req: /// for this constant. pub const PAGE_SIZE: u32 = 4096; +/// Carried by a page's generation counter while some CPU's cache holds +/// lines there (`PpMemSpace::hold_line_page`): that CPU runs the page from +/// its own view, so it is not compiled (`comp::snapshot_page`). It comes off +/// with an ordinary bump, below the held page's last publish, so `publish` +/// takes any change of generation as new bytes. +pub const GEN_HELD: u64 = 1 << 48; + /// Number of possible entry offsets per page: one per 4-byte-aligned word /// (MIPS instructions are always word-aligned), i.e. `PAGE_SIZE / 4` (§2.4: /// `entry_bits` 16×u64 = 1024 bits, `entry_table` 1024 entries). @@ -1620,6 +1627,11 @@ impl PhysicalCodePage { for word in self.denied.iter() { word.store(0, Ordering::Relaxed); } } + /// Undo [`Self::denylist_all`]. + pub fn allow_all(&self) { + for word in self.denied.iter() { word.store(u64::MAX, Ordering::Relaxed); } + } + /// Clear `offset_word`'s `compiled` bit only — NOT `denied`. Two /// distinct callers: /// @@ -1854,8 +1866,9 @@ impl PhysicalCodePage { /// a reader that observes a `compiled` bit set (Acquire-paired via /// `is_runnable`'s `entry_gen` load) never observes a stale/null /// `func` for it. `entry_gen` is written unconditionally on every actual - /// publish, but only **numerically advances** when `snap_gen > - /// entry_gen` (a real page-bump-triggered recompile, §13.6/§13.3's + /// publish, but only **changes** when `snap_gen != entry_gen` (a real + /// page-bump-triggered recompile, or the first after `GEN_HELD` comes + /// off, §13.6/§13.3's /// discussion) — a pure entry-coverage publish against unchanged bytes /// (`snap_gen == entry_gen` already) rewrites the same value. Readers /// never need to distinguish the two: `entry_gen == current_gen()` is @@ -1909,7 +1922,8 @@ impl PhysicalCodePage { } let prev_entry_gen = self.entry_gen.load(Ordering::Relaxed); - let is_real_invalidation = snap_gen > prev_entry_gen; + // Not `>`: a page's generation drops when `GEN_HELD` comes off. + let is_real_invalidation = snap_gen != prev_entry_gen; // Code for the other FR mode replaces the old function outright, so // the old function's entries must go with it: its dispatch switch is // gone, and the new one only has cases for `new_entries`. diff --git a/src/cpu/mips_cache_shadow.rs b/src/cpu/mips_cache_shadow.rs index 7f056a863..8acfcd401 100644 --- a/src/cpu/mips_cache_shadow.rs +++ b/src/cpu/mips_cache_shadow.rs @@ -30,6 +30,18 @@ //! The shadow is deliberately not consulted by `read`/`write`/`fetch`. If it //! ever needs to be, this type is the wrong shape and should say so loudly //! rather than growing a slow path. +//! +//! The R10000-family PROMs use the caches as memory before memory works: they +//! make lines valid with Index Store Tag and Index Store Data and keep a stack +//! or run code in them, while the uncached path sees memory (NEC VR10000 +//! user's manual p. 75). Such a line lives in ppmem's window rather than +//! here: the window this cache reads is the CPU's own cached view +//! (`PpMemSpace::add_cached_view`), and a valid line makes its page private +//! memory in that view, holding the line's bytes +//! ([`ShadowCache::line_enters`]). Loads, stores and fetches read the view as +//! they always did. The CACHE instruction is the only thing that moves a page +//! in or out, and the tags stay the only record of which lines are valid. +//! `clanker/cpu/cache-lines-live-in-the-cpus-view.md` has the limits. use std::cell::UnsafeCell; use std::sync::Arc; @@ -38,6 +50,7 @@ use crate::cpu::mips_cache_v2::{ cache_op_name, model, r4k_load_taglo, r4k_store_taglo, CpuModel, FetchInstrResult, MipsCache, C_ILT, C_IST, C_IWBINV, C_R10K_CBARRIER, C_R10K_ILD, C_R10K_ISD, CACH_PD, CACH_PI, CACH_SD, CACH_SI, }; +use crate::ppmem::PpMemSpace; use crate::traits::{BusDevice, BusRead64, BUS_OK}; use crate::devlog::{LogModule, CACHE_LOG_HIT, CACHE_LOG_OP, devlog_is_active, devlog_mask}; @@ -63,11 +76,15 @@ struct Shadow { /// — see `MRU_BIT`. Hardware state rather than tag storage, which is why /// it is kept here and re-applied on read instead of living in `tags`. mru: Box<[bool]>, + /// The line holds its page in the CPU's cached view + /// (`ShadowCache::line_enters`). + held: Box<[bool]>, } impl Shadow { fn new(lines: usize, data_words: usize) -> Self { Self { + held: vec![false; lines.max(1)].into_boxed_slice(), tags: vec![0u64; lines.max(1)].into_boxed_slice(), data: vec![0u64; data_words].into_boxed_slice(), ecc: vec![0u32; data_words].into_boxed_slice(), @@ -87,6 +104,71 @@ impl Shadow { /// nothing, because the way that is read is never the way that was written. const MRU_BIT: u64 = 1 << 63; +/// R10000-family tag states (NEC U10278EJ3V0UM pp. 285-287): the primary data +/// state in TagLo[7:6] and the secondary state in TagLo[11:10], 0 Invalid. +const PD_STATE_SHIFT: u32 = 6; +const SD_STATE_SHIFT: u32 = 10; + +/// The primary instruction state, TagLo[6] (NEC U10278EJ3V0UM p. 285). +const PI_VALID: u64 = 1 << 6; + +/// A primary instruction word is the instruction predecoded, 36 bits (NEC +/// U10278EJ3V0UM p. 288). Neither manual gives the layout. This one is +/// inferred from the words PROM 6.210 stores, `p` the word and `f` the +/// instruction: p[25:20] = f[5:0], p[19:15] = f[10:6], p[14:10] = f[25:21], +/// p[28:26] = f[28:26], p[35:29] a class that also gives f[31:29], and p[4:0] +/// and p[9:5] the rt and rd fields in the order the class gives. Only the +/// sampled classes decode, and every row is inferred. `addiu $4,$0,2` +/// (0x24040002) is 0x3_4420_0004. +/// +/// Rows: class, f[31:29], whether p[4:0] is rt, the f[28:26] values as a bit +/// mask, and the function fields of the SPECIAL members. +const L1I_CLASSES: &[(u8, u32, bool, u8, &[u8])] = &[ + (0x00, 0, false, 1 << 0, &[0x00]), // nop + (0x08, 0, false, 1 << 0, &[0x38, 0x3c]), // dsll, dsll32 + (0x18, 0, false, 1 << 0, &[0x25]), // or + (0x1a, 1, true, 0b1010_0011, &[]), // addi, addiu, ori, lui + (0x1e, 3, true, 1 << 1, &[]), // daddiu + (0x35, 5, false, 1 << 7, &[]), // cache + (0x36, 6, true, 1 << 7, &[]), // ld + (0x3f, 2, false, 1 << 0, &[]), // mtc0 + (0x40, 0, false, 1 << 3, &[]), // jal + (0x41, 0, false, 0b101, &[0x08]), // jr, j + (0x71, 0, false, 1 << 1, &[]), // REGIMM +]; + +/// The model's own form, bits 35:32 set over an instruction no class holds. +/// Hardware would never produce it. It takes classes 0x78-0x7f, so a real +/// word of those classes would be read as raw. +const L1I_RAW: u64 = 0xf << 32; + +/// The instruction a predecoded word encodes, or the class that is refused. +fn l1i_decode(p: u64) -> Result { + if p >> 32 == L1I_RAW >> 32 { + return Ok(p as u32); + } + let (class, op, funct) = ((p >> 29) as u8 & 0x7f, (p >> 26) as u32 & 7, (p >> 20) as u8 & 0x3f); + let &(_, hi3, rt_low, ops, functs) = L1I_CLASSES.iter().find(|c| c.0 == class).ok_or(class)?; + if ops >> op & 1 == 0 || (op == 0 && !functs.is_empty() && !functs.contains(&funct)) || (class == 0 && p != 0) { + return Err(class); + } + let (low, high) = ((p & 0x1f) as u32, (p >> 5) as u32 & 0x1f); + let (rt, rd) = if rt_low { (low, high) } else { (high, low) }; + Ok(hi3 << 29 | op << 26 | ((p >> 10) as u32 & 0x1f) << 21 | rt << 16 | rd << 11 | ((p >> 15) as u32 & 0x1f) << 6 | funct as u32) +} + +/// The predecoded word for `f`, or the raw form where no class holds it. +fn l1i_encode(f: u32) -> u64 { + let w = f as u64; + let (rt, rd) = (w >> 16 & 0x1f, w >> 11 & 0x1f); + L1I_CLASSES.iter().find_map(|&(class, _, rt_low, _, _)| { + let (low, high) = if rt_low { (rt, rd) } else { (rd, rt) }; + let p = (class as u64) << 29 | (w >> 26 & 7) << 26 | (w & 0x3f) << 20 | (w >> 6 & 0x1f) << 15 + | (w >> 21 & 0x1f) << 10 | high << 5 | low; + (l1i_decode(p) == Ok(f)).then_some(p) + }).unwrap_or(L1I_RAW | w) +} + /// R10000 only: ways per set, as CACHE index operations address them. Bit 0 /// of the index selects the way; the set starts above the line offset. The /// R4400 and R5000 index differently — see `tag_slot`. @@ -154,8 +236,18 @@ pub struct ShadowCache< /// Decoded instructions, by physical page. Never touches the shadow /// arrays above — see `decode_cache`. decode: UnsafeCell, - /// See [`MipsCache::set_pd_tag_store_hook`]. Consulted by `cache_op` only. - pd_tag_hook: UnsafeCell>>, + /// The cached view `tc_base` is, and its number in `PpMemSpace`: where + /// valid lines hold their pages. Null without one. + view_space: UnsafeCell<*const PpMemSpace>, + view: UnsafeCell, + /// For each primary instruction set, the 64-byte half of a held + /// secondary line that a refill there would load (`refilled_word`), or + /// `u64::MAX`. + refill: UnsafeCell>, + /// The held pages whose generation counters are not ppmem's, with the + /// counters carrying their `GEN_HELD`: see [`Self::mark_foreign`]. + #[cfg(feature = "jitv2")] + foreign: UnsafeCell>, } // Safety: the CPU thread is the only accessor, as for every other cache model @@ -212,7 +304,11 @@ impl< #[cfg(feature = "jitv2")] tc_gen: UnsafeCell::new(std::ptr::null_mut()), decode: UnsafeCell::new(crate::cpu::decode_cache::DecodeCache::new()), - pd_tag_hook: UnsafeCell::new(None), + view_space: UnsafeCell::new(std::ptr::null()), + view: UnsafeCell::new(0), + refill: UnsafeCell::new(vec![u64::MAX; Self::PI_SETS].into_boxed_slice()), + #[cfg(feature = "jitv2")] + foreign: UnsafeCell::new(Vec::new()), } } @@ -340,6 +436,278 @@ impl< crate::dlog!(LogModule::L2c, "shadow: -> ISD slot={slot} data={phys_addr:#018x} ecc={:#x}", s.ecc[slot]); } + if s.held.get(self.tag_slot(sel, virt_addr)) == Some(&true) { + self.store_into_held_line(sel, virt_addr, phys_addr); + } + if sel == CACH_PI { + // The set now holds what the guest wrote, not a refill. + if let Some(r) = unsafe { &mut *self.refill.get() }.get_mut((virt_addr as usize / IC_LINE.max(1)) % Self::PI_SETS.max(1)) { + *r = u64::MAX; + } + } + } + + // ---- valid lines hold pages in the CPU's cached view ---- + + /// Primary instruction sets, as Index Load Data (I) addresses them. + const PI_SETS: usize = if IC_LINE == 0 { 0 } else { IC_SIZE / WAYS / IC_LINE }; + + fn view_space(&self) -> Option<&PpMemSpace> { + unsafe { (*self.view_space.get()).as_ref() } + } + + fn view_no(&self) -> usize { + unsafe { *self.view.get() } + } + + /// The physical address and length of the line `tag` names, if it is + /// valid. `va` is an index operand naming the line's slot: the primary + /// caches take PA[11:5] or PA[11:6] from it, the secondary PA[17:7] (NEC + /// U10278EJ3V0UM pp. 285-287; the tag holds the rest). The R5000's + /// primary data tag has the same PTag and PState fields (NEC + /// U11761EJ6V0UM p. 133). + fn line_of(&self, sel: u32, tag: u64, va: u64) -> Option<(u64, usize)> { + let pa_hi = ((tag >> 32) & 0xf) << 36; + match sel { + CACH_PD if DC_LINE != 0 && (tag >> PD_STATE_SHIFT) & 3 != 0 => { + Some((pa_hi | ((tag >> 8) & 0xff_ffff) << 12 | (va & 0xfff & !(DC_LINE as u64 - 1)), DC_LINE)) + } + CACH_PI if IC_LINE != 0 && tag & PI_VALID != 0 => { + Some((pa_hi | ((tag >> 8) & 0xff_ffff) << 12 | (va & 0xfff & !(IC_LINE as u64 - 1)), IC_LINE)) + } + CACH_SD if L2_LINE != 0 && (tag >> SD_STATE_SHIFT) & 3 != 0 => { + Some((pa_hi | ((tag >> 14) & 0x3_ffff) << 18 | (va & 0x3_ffff & !(L2_LINE as u64 - 1)), L2_LINE)) + } + _ => None, + } + } + + /// An index operand that names slot `i` of `sel`: `tag_slot`'s inverse. + fn slot_va(&self, sel: u32, i: usize) -> u64 { + let line = match sel { CACH_PI => IC_LINE, CACH_PD => DC_LINE, _ => L2_LINE }; + if Self::R5K && matches!(sel, CACH_PI | CACH_PD) { + let sets = if sel == CACH_PI { Self::IC_LINES } else { Self::DC_LINES } / 2; + return ((i / sets) << 14 | (i % sets) * line) as u64; + } + ((i / WAYS) * line + (i % WAYS)) as u64 + } + + /// A CACHE instruction changed slot `idx` of `sel`'s tag from `old` to + /// `new`; `va` is the operand that named it. A held line that stops being + /// valid, or moves, gives its page back; one that becomes valid takes one. + /// The tags stay the only record of which lines are valid. + #[cold] + #[inline(never)] + fn tag_changes(&self, sel: u32, idx: usize, va: u64, old: u64, new: u64) { + let Some(sp) = self.view_space() else { return }; + let now = self.line_of(sel, new, va); + if self.shadow(sel).held[idx] { + let was = self.line_of(sel, old, va).expect("a held line is valid"); + if now == Some(was) { + return; // the state changed, not the line + } + self.line_leaves(sp, sel, idx, was); + } + if let Some(line) = now { + self.line_enters(sp, sel, idx, va, line); + } + } + + /// Slot `idx` of `sel` holds a valid line for `[pa, pa + len)`: its page + /// becomes private in the CPU's view and the line's bytes are written + /// into it from the data array, since Index Store Tag writes the tag + /// alone (NEC VR5000/VR10000 instruction manual pp. 24-25). A line with + /// no memory behind it holds no page and stays in the arrays, as every + /// line did before, unless the machine allows lines there. + fn line_enters(&self, sp: &PpMemSpace, sel: u32, idx: usize, va: u64, (pa, len): (u64, usize)) { + let unbacked = !sp.backs(pa); + let Some(new) = sp.hold_line_page(self.view_no(), pa) else { return }; + let base = unsafe { *self.tc_base.get() }; + if new { + self.mark_foreign(sp, pa, true); + } + if new && unbacked { + // No memory behind the page yet: it starts as what the cache's + // bus shows there. + let gran = crate::ppmem::granularity() as u64; + for a in (pa & !(gran - 1)..(pa & !(gran - 1)) + gran).step_by(8) { + let r = self.downstream.read64(a); + unsafe { *(base.add(a as usize) as *mut u64) = if r.is_ok() { r.data.rotate_left(32) } else { 0 } }; + } + } + if l2_op_log(CACHE_LOG_OP) { + crate::dlog!(LogModule::L2c, "shadow: -> line {pa:#x} holds its page (new: {new})"); + } + let s = self.shadow(sel); + s.held[idx] = true; + if s.data.is_empty() { + return; // the R5000 has no Index Store Data: the page starts as memory + } + let (way, first) = (va & 1, va & !(len as u64 - 1)); + let unit = if sel == CACH_SD { 8 } else { 4 }; + for k in 0..len / unit { + let word = s.data[self.data_slot(sel, s.data.len(), first + (unit * k) as u64 + way)]; + let at = pa + (unit * k) as u64; + match sel { + CACH_PI => self.put_instruction(at, word), + CACH_PD => unsafe { *(base.add(at as usize) as *mut u32) = word as u32 }, + // TagHi is the doubleword's high word and TagLo its low word + // (MIPS R10000 UM p. 201; NEC VR10000 UM p. 289, Fig. 13-33). + // Neither manual says which address each word belongs to. + // TagLo at the lower address is inferred from shipped PROMs: + // PROM 6.210 stages code pairs with the word at A in TagLo + // and the one at A + 4 in TagHi, and its staged code equals + // its image only this way, as does the IP35 boot-flash + // PROM's staging. The window keeps a doubleword's words in + // address order, low half first, so this is the data as is. + _ => unsafe { *(base.add(at as usize) as *mut u64) = word }, + } + } + if sel == CACH_SD { + self.set_refill(s.tags[idx], pa, len, true); + } + } + + /// The line in slot `idx` of `sel` leaves, unwritten, as Index Store Tag + /// leaves it. With the page's last line the view shows memory there + /// again. Until then the line's bytes stay in the page. + fn line_leaves(&self, sp: &PpMemSpace, sel: u32, idx: usize, (pa, len): (u64, usize)) { + let s = self.shadow(sel); + s.held[idx] = false; + if sel == CACH_SD { + self.set_refill(s.tags[idx], pa, len, false); + } + let gone = sp.release_line_page(self.view_no(), pa); + if gone { + self.mark_foreign(sp, pa, false); + } + if l2_op_log(CACHE_LOG_OP) { + crate::dlog!(LogModule::L2c, "shadow: -> line {pa:#x} leaves (page released: {gone})"); + } + } + + /// ppmem puts `GEN_HELD` on its own generation counters. Where the cache's + /// bus counts a held page's writes on a counter of its own, which compiles + /// check instead (the Fuel's reset PROM, mapped over memory), the mark + /// goes there too, and comes off with the page's last line. A page + /// outside any counter, or on ppmem's, is skipped. + #[cfg_attr(not(feature = "jitv2"), allow(unused_variables))] + fn mark_foreign(&self, sp: &PpMemSpace, pa: u64, held: bool) { + #[cfg(feature = "jitv2")] + { + use crate::cpu::jitv2::jitv2::GEN_HELD; + use crate::ppmem::MappedMemory; + use std::sync::atomic::{AtomicU64, Ordering}; + let page = pa & !(crate::ppmem::granularity() as u64 - 1); + let foreign = unsafe { &mut *self.foreign.get() }; + if held { + for a in (page..page + crate::ppmem::granularity() as u64).step_by(4096) { + let g = self.downstream.gen_ptr(a); + let own = unsafe { sp.gen_window_base().add((a >> 12) as usize) as *const AtomicU64 }; + if !g.is_null() && g != own { + unsafe { (*g).fetch_add(GEN_HELD, Ordering::Release) }; + foreign.push((page, g)); + } + } + } else { + foreign.retain(|&(p, g)| { + if p == page { + unsafe { (*g).fetch_add(1u64.wrapping_sub(GEN_HELD), Ordering::Release) }; + } + p != page + }); + } + } + } + + /// Index Store Data into a held line writes through to its page, so the + /// next cached access sees it. Same quadword rule as the data array. + fn store_into_held_line(&self, sel: u32, va: u64, value: u64) { + let idx = self.tag_slot(sel, va); + let Some((pa, len)) = self.line_of(sel, self.shadow(sel).tags[idx], va) else { return }; + let unit = if sel == CACH_SD { 8 } else { 4 }; + let at = pa + (va & (len as u64 - 1) & !(unit - 1)); + let base = unsafe { *self.tc_base.get() }; + match sel { + CACH_PI => self.put_instruction(at, value & 0xf_ffff_ffff), + CACH_PD => unsafe { *(base.add(at as usize) as *mut u32) = value as u32 }, + _ => unsafe { + *(base.add(at as usize) as *mut u64) = value; + *(base.add((at ^ 8) as usize) as *mut u64) = 0; + }, + } + if sel == CACH_SD { + self.set_refill(self.shadow(sel).tags[idx], pa, len, true); + } + } + + /// A primary instruction word goes into its line's page as the + /// instruction it predecodes, so a fetch runs it. A word of a class the + /// model does not know leaves the page as it was, where hardware would + /// run it. + fn put_instruction(&self, at: u64, word: u64) { + match l1i_decode(word) { + Ok(f) => unsafe { *((*self.tc_base.get()).add(at as usize) as *mut u32) = f }, + Err(class) => crate::dlog!(LogModule::L2c, + "shadow: primary instruction word {word:#011x} at {at:#x}: predecode class {class:#04x} unknown, memory's word runs"), + } + } + + /// A held secondary line `[pa, pa + len)` with tag `tag` is what a primary + /// instruction refill in each set it maps to would load, or stops being. + /// The set is the line's virtual index, TagLo[8:7] (VA[13:12], NEC + /// U10278EJ3V0UM p. 287), over PA[11:6]. Two lines for one set: the + /// later wins. + fn set_refill(&self, tag: u64, pa: u64, len: usize, held: bool) { + let refill = unsafe { &mut *self.refill.get() }; + if !held { + // By address: the tag may already name another line. + refill.iter_mut().filter(|r| (pa..pa + len as u64).contains(*r)).for_each(|r| *r = u64::MAX); + return; + } + for half in (pa..pa + len as u64).step_by(IC_LINE.max(1)) { + if let Some(r) = refill.get_mut(((tag >> 7 & 3) << 12 | (half & 0xfff)) as usize / IC_LINE) { + *r = half; + } + } + } + + /// Index Load Data (I) in a set a held secondary line maps to, which + /// Index Store Data (I) has not written since and where no held primary + /// line is: the word a refill from that + /// line would have left, predecoded. The model never refills on a fetch, + /// so this assumes the line was refilled, whichever way the operand + /// names. The Tezro's PROM runs a staged secondary line and reads the + /// primary words back to stage them itself. Index Load Tag (I) still + /// reads the stored tag. + #[cold] + #[inline(never)] + fn refilled_word(&self, va: u64) -> Option { + let half = *unsafe { &*self.refill.get() }.get((va as usize / IC_LINE.max(1)) % Self::PI_SETS.max(1))?; + if half == u64::MAX || self.shadow(CACH_PI).held[self.tag_slot(CACH_PI, va)] { + return None; + } + let at = half + (va & (IC_LINE as u64 - 1) & !3); + Some(l1i_encode(unsafe { *((*self.tc_base.get()).add(at as usize) as *const u32) })) + } + + /// How many lines hold pages. + fn held_count(&self) -> usize { + [CACH_PI, CACH_PD, CACH_SD].iter().map(|&sel| self.shadow(sel).held.iter().filter(|&&h| h).count()).sum() + } + + /// Every held line goes, with no writeback: power-on, or a snapshot + /// restore. + fn drop_held_lines(&self) { + let Some(sp) = self.view_space() else { return }; + for sel in [CACH_PI, CACH_PD, CACH_SD] { + for i in 0..self.shadow(sel).held.len() { + if self.shadow(sel).held[i] { + let line = self.line_of(sel, self.shadow(sel).tags[i], self.slot_va(sel, i)).expect("a held line is valid"); + self.line_leaves(sp, sel, i, line); + } + } + } } } @@ -417,8 +785,19 @@ impl< fn tcache_base_ptr(&self) -> *mut u8 { unsafe { *self.tc_base.get() } } - fn set_pd_tag_store_hook(&self, hook: Box) { - unsafe { *self.pd_tag_hook.get() = Some(hook) }; + unsafe fn set_cached_view(&self, sp: *const PpMemSpace, view: usize) { + unsafe { + *self.view_space.get() = sp; + *self.view.get() = view; + } + } + + /// `l2 probe`: the lines holding pages in the CPU's cached view. + fn debug_probe(&self, _cache_name: &str, _virt_addr: u64, _phys_addr: u64) -> String { + let pages = self.view_space().map(|sp| sp.held_line_pages(self.view_no())).unwrap_or_default(); + let list: Vec = pages.iter().map(|p| format!("{p:#x}")).collect(); + format!("shadow: {} valid lines hold {} pages in the cached view [{}]", + self.held_count(), pages.len(), list.join(" ")) } /// Tags only: a fetch reads memory through the decode cache and changes @@ -565,12 +944,6 @@ impl< let sel = cache_op & 3; let op = cache_op & 0x1C; - if op == C_IST && sel == CACH_PD { - if let Some(hook) = unsafe { &*self.pd_tag_hook.get() } { - hook(phys_addr as u32); - } - } - // Tag operations under `op`, the whole data-array walk only under // `hit` as well — see `l2_op_log`. if l2_op_log(if matches!(op, C_IST | C_ILT) { CACHE_LOG_OP } else { CACHE_LOG_HIT }) { @@ -589,7 +962,13 @@ impl< let idx = self.tag_slot(sel, virt_addr); let s = self.shadow(sel); if op == C_IST { - s.tags[idx] = r4k_store_taglo(Self::R5K, sel, phys_addr as u32) as u64; + let old = std::mem::replace(&mut s.tags[idx], r4k_store_taglo(Self::R5K, sel, phys_addr as u32) as u64); + // The O2's POST keeps its stack in R5000 primary data lines + // as it does in R10000 ones. An instruction line, with no + // Index Store Data to fill it, could hold only memory. + if Self::R5K && sel == CACH_PD && old != s.tags[idx] { + self.tag_changes(sel, idx, virt_addr, old, s.tags[idx]); + } return 0; } return r4k_load_taglo(Self::R5K, sel, s.tags[idx] as u32) as u64; @@ -619,7 +998,11 @@ impl< // here destroyed real tag bits. let set = (idx / WAYS).min(s.mru.len() - 1); s.mru[set] = phys_addr & MRU_BIT != 0; + let old = s.tags[idx]; s.tags[idx] = phys_addr & mask & !MRU_BIT; + if Self::R10K && matches!(sel, CACH_PI | CACH_PD | CACH_SD) && old != s.tags[idx] { + self.tag_changes(sel, idx, virt_addr, old, s.tags[idx]); + } } 0 } @@ -654,6 +1037,11 @@ impl< crate::dlog!(LogModule::L2c, "shadow: -> ILD slot={slot} data={:#018x} ecc={:#x}", s.data[slot], s.ecc[slot]); } + if sel == CACH_PI { + if let Some(p) = self.refilled_word(virt_addr) { + return p; + } + } s.data[slot] } } @@ -666,7 +1054,8 @@ impl< let idx = self.tag_slot(sel, virt_addr); let s = self.shadow(sel); if idx < s.tags.len() { - s.tags[idx] = 0; + let old = std::mem::take(&mut s.tags[idx]); + self.tag_changes(sel, idx, virt_addr, old, 0); } 0 } @@ -680,6 +1069,7 @@ impl< /// Power-on: tags read back invalid, and nothing decoded survives — RAM /// may be reloaded without a write counter moving. fn power_on(&self) { + self.drop_held_lines(); for sel in [CACH_PI, CACH_PD, CACH_SD] { let s = self.shadow(sel); s.tags.fill(0); @@ -688,10 +1078,27 @@ impl< unsafe { (*self.decode.get()).flush() }; } - /// A restore replaces RAM wholesale; drop every decoded instruction. The - /// shadow arrays are not snapshotted (nothing reads them but diagnostics). - fn load_cache_state(&self, _v: &toml::Value) -> Result<(), String> { + /// The shadow arrays are not snapshotted (nothing reads them but + /// diagnostics). Lines holding pages are, as a count: a PROM's cache + /// phase is the only time any are held, and a snapshot of it would + /// restore without them, so it is refused. + fn save_cache_state(&self) -> toml::Value { + let mut t = toml::map::Map::new(); + let n = self.held_count(); + if n > 0 { + t.insert("held_lines".into(), toml::Value::Integer(n as i64)); + } + toml::Value::Table(t) + } + + /// A restore replaces RAM wholesale; drop every decoded instruction and + /// every held line. + fn load_cache_state(&self, v: &toml::Value) -> Result<(), String> { + if let Some(n) = crate::snapshot::get_field(v, "held_lines") { + return Err(format!("cache: the snapshot was taken with {n} cache lines holding memory, which it does not carry")); + } unsafe { (*self.decode.get()).flush() }; + self.drop_held_lines(); Ok(()) } @@ -795,7 +1202,6 @@ pub type R12000Module4MShadowCache = pub type R12000O2ShadowCache = ShadowCache<32768, 64, 32768, 32, 2097152, 64, true, 0x0000_0e35, 0x0000_0900, 64, { model::R12000 }>; - /// SGI O2 R10000 (IP32): the IP28 model with 64-byte secondary-cache lines. /// /// The O2 PROM (4.18 and earlier) handles only that line size. With Config's @@ -1084,20 +1490,6 @@ mod tests { (::PRID, ::FIR, ::MIPS4, ::TLB_ENTRIES, ::VA_BITS, ::R10K_CACHE_OPS)); } - /// The tag store hook sees primary data tag stores and nothing else. - #[test] - fn the_tag_store_hook_sees_only_primary_data_tag_stores() { - let c = cache(); - let seen = Arc::new(std::sync::Mutex::new(Vec::new())); - let s = seen.clone(); - c.set_pd_tag_store_hook(Box::new(move |tag| s.lock().unwrap().push(tag))); - c.cache_op(C_IST | CACH_PI, 0x8000_0000, 0x11); - c.cache_op(C_IST | CACH_SD, 0x8000_0000, 0x22); - c.cache_op(C_ILT | CACH_PD, 0x8000_0000, 0x33); - c.cache_op(C_IST | CACH_PD, 0x8000_0000, 0xc8); - assert_eq!(*seen.lock().unwrap(), vec![0xc8]); - } - /// Index_Store_Tag then Index_Load_Tag returns exactly what the full model /// returns, for random TagLo words at random indexes: the shadow must not /// be observably different to a PROM, IDE or IRIX reading tags. Both @@ -1144,4 +1536,356 @@ mod tests { assert!(g.supported && g.tagless && !g.has_l2, "tagless, and no L2 decode slots for a store to invalidate: {g:?}"); } + + // ---- valid lines hold pages in the CPU's cached view ---- + + use crate::ppmem::{MappedMemory, PpMemory}; + + /// A cache reading memory through a cached view of a fresh window, with + /// a 64 MB bank mapped at 0. Its bus is the bank itself: memory as the + /// uncached path sees it. Boxed: the view publishes into the cache's + /// inline bitmap field. + fn windowed() -> (Box, Box, Arc) { + let (space, mut banks) = PpMemSpace::with_bank_sizes(&[64]).unwrap(); + let space = Box::new(space); + let bank = Arc::new(banks.remove(0)); + let c = Box::new(R10000ShadowCache::from(bank.clone() as Arc)); + attach(&c, &space); + space.map_bank(0, 0, 64 << 20, 64 << 20).unwrap(); + (c, space, bank) + } + + fn attach(c: &R10000ShadowCache, space: &PpMemSpace) -> usize { + let v = space.add_cached_view(&[c.tcache_bitmap_ptr()]).unwrap(); + unsafe { + c.set_tcache_window(space.view_base(v)); + c.set_cached_view(space, v); + } + v + } + + /// TagLo of a DirtyExclusive primary data line for `pa`'s page. + fn pd_dirty(pa: u64) -> u64 { (pa >> 12) << 8 | 3 << PD_STATE_SHIFT } + + /// TagLo of a secondary line for `pa` in `state`. + fn sd_tag(pa: u64, state: u64) -> u64 { (pa >> 18) << 14 | state << SD_STATE_SHIFT } + + /// The O2's POST: a valid tag for page 0, the page tested uncached while + /// the cached side keeps its stack, and an invalid tag that drops the + /// line without a writeback. + #[test] + fn a_valid_line_gives_the_cached_side_its_own_page() { + let (c, space, bank) = windowed(); + bank.write32(0x40, 0x1111_1111); + c.cache_op(C_R10K_ISD | CACH_PD, 0x8000_0040, 0x1111_1111); + c.cache_op(C_IST | CACH_PD, 0x8000_0040, pd_dirty(0)); + assert_eq!(space.held_line_pages(0), vec![0]); + assert_eq!(c.read::<4>(0, 0x40).data, 0x1111_1111); + c.write::<4>(0, 0x40, 0x2222_2222); + bank.write32(0x40, 0x3333_3333); + assert_eq!(c.read::<4>(0, 0x40).data, 0x2222_2222, "cached: the line"); + assert_eq!(bank.read32(0x40).data, 0x3333_3333, "uncached: memory"); + c.cache_op(C_IST | CACH_PD, 0x8000_0040, 0); + assert!(space.held_line_pages(0).is_empty()); + assert_eq!(c.read::<4>(0, 0x40).data, 0x3333_3333, "dropped, not written back"); + } + + /// Index Store Tag writes the tag alone, so the line holds what Index + /// Store Data put in the data array before, and Index Store Data after + /// writes through. Index Load Data reads the array. + #[test] + fn a_line_holds_its_data_array_and_takes_later_stores() { + let (c, _space, bank) = windowed(); + bank.write32(0x2004, 0x5555_5555); + c.cache_op(C_R10K_ISD | CACH_PD, 0x8000_0004, 0xdead_beef); + c.cache_op(C_IST | CACH_PD, 0x8000_0000, pd_dirty(0x2000)); + assert_eq!(c.read::<4>(0, 0x2004).data, 0xdead_beef); + c.cache_op(C_R10K_ISD | CACH_PD, 0x8000_000c, 0x0123_4567); + assert_eq!(c.read::<4>(0, 0x200c).data, 0x0123_4567); + assert_eq!(c.cache_op(C_R10K_ILD | CACH_PD, 0x8000_000c, 0), 0x0123_4567); + assert_eq!(bank.read32(0x2004).data, 0x5555_5555, "memory never saw the line"); + } + + /// A secondary line takes a doubleword at a time with TagLo at the lower + /// address. That is how the shipped PROMs stage code: the instruction at + /// A in TagLo and the one at A + 4 in TagHi, then Index Store Data (S), + /// and cached accesses see them in that order. Index Store Data (S) also + /// zeroes the other doubleword of its quadword (R10000 UM p. 202). + #[test] + fn a_secondary_line_runs_code_staged_the_way_the_proms_stage_it() { + let (c, space, _bank) = windowed(); + let (at_a, at_a4) = (0x2404_0002u64, 0x03e0_0008u64); // addiu $4,$0,2; jr $31 + c.cache_op(C_R10K_ISD | CACH_SD, 0x8000_3008, at_a4 << 32 | at_a); + c.cache_op(C_IST | CACH_SD, 0x8000_3000, sd_tag(0x3000, 2)); + assert_eq!(space.held_line_pages(0), vec![0x3000]); + assert_eq!(c.read::<4>(0, 0x3008).data, at_a, "the word at A"); + assert_eq!(c.read::<4>(0, 0x300c).data, at_a4, "the word at A + 4"); + c.cache_op(C_R10K_ISD | CACH_SD, 0x8000_3000, 0x9999); + assert_eq!(c.read::<8>(0, 0x3000).data, 0x9999 << 32); + assert_eq!(c.read::<8>(0, 0x3008).data, 0, "the partner doubleword is zeroed"); + } + + /// A tag test's patterns name no memory: they hold nothing, and stay + /// in the arrays as before. + #[test] + fn a_line_with_no_memory_behind_holds_no_page() { + let (c, space, _bank) = windowed(); + c.cache_op(C_IST | CACH_SD, 0x8000_0000, sd_tag(0x55_5554_0000, 3) | 0x5); + c.cache_op(C_IST | CACH_PD, 0x8000_0000, pd_dirty(0x8000_0000)); + assert!(space.held_line_pages(0).is_empty()); + assert_eq!(c.cache_op(C_ILT | CACH_SD, 0x8000_0000, 0) & 0xffff_ffff, (sd_tag(0x55_5554_0000, 3) | 0x5) & 0xffff_ffff); + } + + /// The R4400 PROMs keep nothing in their caches: a valid tag there holds + /// no page, and none of this runs for them. + #[test] + fn an_r4400_tag_holds_nothing() { + let (space, mut banks) = PpMemSpace::with_bank_sizes(&[64]).unwrap(); + let bank = Arc::new(banks.remove(0)); + let c = Box::new(R4400ShadowCache::from(bank as Arc)); + let v = space.add_cached_view(&[c.tcache_bitmap_ptr()]).unwrap(); + unsafe { + c.set_tcache_window(space.view_base(v)); + c.set_cached_view(&space, v); + } + space.map_bank(0, 0, 64 << 20, 64 << 20).unwrap(); + c.cache_op(C_IST | CACH_PD, 0x8000_2000, 0x2 << 8 | 3 << 6); + c.cache_op(C_IST | CACH_SD, 0x8000_2000, 0x1 << 13 | 5 << 10); + assert!(space.held_line_pages(v).is_empty()); + } + + /// An R5000 cache on a cached view of a fresh window with a 64 MB bank + /// at 0, as `windowed` builds an R10000 one. + fn r5000_windowed() -> (Box, Box, Arc) { + let (space, mut banks) = PpMemSpace::with_bank_sizes(&[64]).unwrap(); + let space = Box::new(space); + let bank = Arc::new(banks.remove(0)); + let c = Box::new(R5000ShadowCache::from(bank.clone() as Arc)); + let v = space.add_cached_view(&[c.tcache_bitmap_ptr()]).unwrap(); + unsafe { + c.set_tcache_window(space.view_base(v)); + c.set_cached_view(&*space, v); + } + space.map_bank(0, 0, 64 << 20, 64 << 20).unwrap(); + (c, space, bank) + } + + /// The O2's POST on the R5000, as the shipped PROM runs it on iris: + /// Index Store Tag (D) with TagLo 0xc0 (page 0, Valid) at every index of + /// 0x8000_0000..=0x8000_1000, the stack stored cached, page 0 tested + /// uncached, then an invalid tag at each index. The cached side keeps + /// the stack until the last line goes, which drops it unwritten. The + /// R5000 has no Index Store Data, so the page starts as memory. + #[test] + fn an_r5000_primary_data_line_gives_the_cached_side_its_own_page() { + let (c, space, bank) = r5000_windowed(); + bank.write32(0xff4, 0x1111_1111); + for va in (0x8000_0000..=0x8000_1000u64).step_by(32) { + c.cache_op(C_IST | CACH_PD, va, 0xc0); + } + assert_eq!(space.held_line_pages(0), vec![0]); + assert_eq!(c.read::<4>(0, 0xff4).data, 0x1111_1111, "the page starts as memory"); + c.write::<4>(0, 0xff4, 0xbfc0_4400); + bank.write32(0xff4, 0xa000_0ff0); + assert_eq!(c.read::<4>(0, 0xff4).data, 0xbfc0_4400, "cached: the stack"); + assert_eq!(bank.read32(0xff4).data, 0xa000_0ff0, "uncached: the memory test"); + assert_eq!(c.cache_op(C_ILT | CACH_PD, 0x8000_0fe0, 0) & 0xc0, 0xc0, "the tag reads back"); + for va in (0x8000_0020..=0x8000_1000u64).step_by(32) { + c.cache_op(C_IST | CACH_PD, va, 0); + } + assert_eq!(space.held_line_pages(0), vec![0], "until its last line goes"); + c.cache_op(C_IST | CACH_PD, 0x8000_0000, 0); + assert!(space.held_line_pages(0).is_empty()); + assert_eq!(c.read::<4>(0, 0xff4).data, 0xa000_0ff0, "dropped, not written back"); + } + + /// The R5000's second way, VA[14], holds a page like the first, and + /// power-on drops it. Instruction lines hold nothing: with no Index + /// Store Data, one could hold only what memory holds. There is no + /// secondary. + #[test] + fn an_r5000_line_in_either_way_holds_and_power_on_drops_it() { + let (c, space, _bank) = r5000_windowed(); + c.cache_op(C_IST | CACH_PI, 0x8000_0000, 0x3 << 8 | 0x80); + c.cache_op(C_IST | CACH_SD, 0x8000_0000, 0x1 << 13 | 5 << 10); + assert!(space.held_line_pages(0).is_empty()); + c.cache_op(C_IST | CACH_PD, 0x8000_4fe0, 0x5 << 8 | 0xc0); + assert_eq!(space.held_line_pages(0), vec![0x5000]); + assert_eq!(c.cache_op(C_ILT | CACH_PD, 0x8000_0fe0, 0) & 0xc0, 0, "way 0 is untouched"); + c.power_on(); + assert!(space.held_line_pages(0).is_empty()); + assert!(c.debug_probe("l2", 0, 0).contains("0 valid lines hold 0 pages")); + } + + /// The Tezro: two CPUs keep their stacks in lines for the same physical + /// page, with different contents, and neither sees the other's. + #[test] + fn two_cpus_hold_the_same_page_apart() { + let (space, mut banks) = PpMemSpace::with_bank_sizes(&[64]).unwrap(); + let space = Box::new(space); + let bank = Arc::new(banks.remove(0)); + let a = Box::new(R10000ShadowCache::from(bank.clone() as Arc)); + let b = Box::new(R10000ShadowCache::from(bank.clone() as Arc)); + attach(&a, &space); + attach(&b, &space); + space.map_bank(0, 0, 64 << 20, 64 << 20).unwrap(); + bank.write32(0x10_0040, 0x1234_5678); + for c in [&a, &b] { + c.cache_op(C_R10K_ISD | CACH_PD, 0x8000_0040, 0x1234_5678); + c.cache_op(C_IST | CACH_PD, 0x8000_0040, pd_dirty(0x10_0000)); + } + a.write::<4>(0, 0x10_0040, 0xaaaa_aaaa); + b.write::<4>(0, 0x10_0040, 0xbbbb_bbbb); + assert_eq!(a.read::<4>(0, 0x10_0040).data, 0xaaaa_aaaa); + assert_eq!(b.read::<4>(0, 0x10_0040).data, 0xbbbb_bbbb); + assert_eq!(bank.read32(0x10_0040).data, 0x1234_5678, "memory is neither"); + assert_eq!(unsafe { *(space.window_base().add(0x10_0040) as *const u32) }, 0x1234_5678, + "the shared window, which other masters read, is memory"); + a.cache_op(C_IST | CACH_PD, 0x8000_0040, 0); + assert_eq!(a.read::<4>(0, 0x10_0040).data, 0x1234_5678); + assert_eq!(b.read::<4>(0, 0x10_0040).data, 0xbbbb_bbbb); + } + + /// Two lines of one page: the page stays until the last goes. The R12000 + /// zeroing a primary data tag at Index Writeback Invalidate ends a line + /// like an invalid tag store. + #[test] + fn a_page_goes_with_its_last_line() { + let (space, mut banks) = PpMemSpace::with_bank_sizes(&[64]).unwrap(); + let space = Box::new(space); + let bank = Arc::new(banks.remove(0)); + let c = Box::new(R12000ShadowCache::from(bank.clone() as Arc)); + let v = space.add_cached_view(&[c.tcache_bitmap_ptr()]).unwrap(); + unsafe { + c.set_tcache_window(space.view_base(v)); + c.set_cached_view(&*space, v); + } + space.map_bank(0, 0, 64 << 20, 64 << 20).unwrap(); + bank.write32(0x6000, 0x600d); + c.cache_op(C_IST | CACH_PD, 0x8000_6000, pd_dirty(0x6000)); + c.cache_op(C_IST | CACH_PD, 0x8000_6040, pd_dirty(0x6000)); + c.write::<4>(0, 0x6000, 0xbad); + c.cache_op(C_IST | CACH_PD, 0x8000_6000, 0); + assert_eq!(space.held_line_pages(v), vec![0x6000]); + assert_eq!(c.read::<4>(0, 0x6000).data, 0xbad, "the page stays the view's"); + c.cache_op(C_IWBINV | CACH_PD, 0x8000_6040, 0); + assert!(space.held_line_pages(v).is_empty()); + assert_eq!(c.read::<4>(0, 0x6000).data, 0x600d); + } + + /// Power-on drops every held page. A snapshot taken while lines hold + /// pages is refused at restore. + #[test] + fn power_on_drops_held_pages_and_a_snapshot_of_them_is_refused() { + let (c, space, _bank) = windowed(); + c.cache_op(C_IST | CACH_PD, 0x8000_0000, pd_dirty(0x7000)); + let saved = c.save_cache_state(); + assert!(c.load_cache_state(&saved).is_err()); + c.power_on(); + assert!(space.held_line_pages(0).is_empty()); + assert!(c.debug_probe("l2", 0, 0).contains("0 valid lines hold 0 pages")); + assert!(c.load_cache_state(&c.save_cache_state()).is_ok()); + } + + // ---- primary instruction lines ---- + + /// Every class of the table, encoded and decoded back, one field moved + /// at a time for the example word, and what the table refuses. + #[test] + fn predecoded_words_round_trip() { + let p = 0x3_4420_0004; // addiu $4,$0,2 + for (q, f) in [(p, 0x2404_0002), (p | 0x1f << 5, 0x2404_f802), (p | 0x1f << 10, 0x27e4_0002), (p | 0x1f << 15, 0x2404_07c2)] { + assert_eq!(l1i_decode(q), Ok(f), "{q:#x}"); + } + for (class, f) in [(0x08, 0x0009_20f8), (0x08, 0x0009_20fc), (0x18, 0x0109_5025), (0x1a, 0x2084_0010), + (0x1a, 0x3484_0010), (0x1a, 0x3c04_8000), (0x1e, 0x6484_0010), (0x35, 0xbd04_0040), + (0x36, 0xdd04_0008), (0x3f, 0x4084_6000), (0x40, 0x0c00_1234), (0x41, 0x03e0_0008), + (0x41, 0x0800_1234), (0x71, 0x0481_0004), (0, 0)] { + assert_eq!((l1i_encode(f) >> 29, l1i_decode(l1i_encode(f))), (class, Ok(f)), "{f:#x}"); + } + for f in [0x0009_20c0, 0x2884_0010, 0x4422_3000] { // sll, slti, mtc1: no class + assert_eq!((l1i_encode(f), l1i_decode(L1I_RAW | f as u64)), (L1I_RAW | f as u64, Ok(f))); + } + assert_eq!((l1i_decode(1), l1i_decode(0x2a << 29)), (Err(0), Err(0x2a))); + } + + /// TagLo of a valid primary instruction line for `pa`'s page. + fn pi_valid(pa: u64) -> u64 { (pa >> 12) << 8 | PI_VALID } + + /// A valid primary instruction line puts the instructions its words + /// predecode into the page, leaving memory's word for one it cannot + /// decode, and Index Store Data (I) writes through. + #[test] + fn a_primary_instruction_line_puts_its_instructions_in_the_page() { + let (c, space, bank) = windowed(); + bank.write32(0x9008, 0x1111_1111); + let va = 0x8000_9001u64; // way 1 + c.cache_op(C_R10K_ISD | CACH_PI, va, 0x3_4420_0004); + c.cache_op(C_R10K_ISD | CACH_PI, va + 8, 0x2a << 29); + c.cache_op(C_IST | CACH_PI, va, pi_valid(0x9000)); + assert_eq!(space.held_line_pages(0), vec![0x9000]); + assert_eq!([0x9000, 0x9008].map(|a| c.read::<4>(0, a).data), [0x2404_0002, 0x1111_1111]); + c.cache_op(C_R10K_ISD | CACH_PI, va + 4, l1i_encode(0x03e0_0008)); + assert_eq!(c.read::<4>(0, 0x9004).data, 0x03e0_0008); + assert_eq!(c.cache_op(C_R10K_ILD | CACH_PI, va, 0), 0x3_4420_0004, "Index Load Data (I) reads the array"); + assert_eq!(bank.read32(0x9000).data, 0, "memory never saw it"); + c.cache_op(C_IST | CACH_PI, va, 0); + assert!(space.held_line_pages(0).is_empty()); + } + + /// A held secondary line answers Index Load Data (I) in the primary sets + /// it maps to, as the predecoded words a refill would leave, until Index + /// Store Data (I) writes the set or the line goes. A held primary line + /// answers for itself. + #[test] + fn index_load_data_i_reads_a_refill_until_the_guest_writes_the_set() { + let (c, _space, _bank) = windowed(); + c.cache_op(C_IST | CACH_SD, 0x8000_4f00, sd_tag(0x4f00, 2) | 1 << 7); // VA[13:12] = 1 + c.cache_op(C_R10K_ISD | CACH_SD, 0x8000_4f10, 0x0140_0008); // jr $10 at 0x4f10 + let set = |off: u64| 0x8000_1f00 | off; + for va in [set(0x10), set(0x10) | 1] { + assert_eq!(c.cache_op(C_R10K_ILD | CACH_PI, va, 0), l1i_encode(0x0140_0008), "either way"); + } + assert_eq!(c.cache_op(C_R10K_ILD | CACH_PI, 0x8000_0f10, 0), 0, "another set: the array"); + c.cache_op(C_R10K_ISD | CACH_PI, set(0x10), 0x5_5555_5555); + assert_eq!(c.cache_op(C_R10K_ILD | CACH_PI, set(0x10), 0), 0x5_5555_5555, "the guest's pattern"); + c.cache_op(C_R10K_ISD | CACH_SD, 0x8000_4f10, 0x0140_0008); + assert_eq!(c.cache_op(C_R10K_ILD | CACH_PI, set(0x10), 0), l1i_encode(0x0140_0008), "restaged: a refill again"); + c.cache_op(C_R10K_ISD | CACH_PI, set(0x10), 0x3_4420_0004); + c.cache_op(C_IST | CACH_PI, set(0x10), pi_valid(0x1_4000)); + c.cache_op(C_R10K_ISD | CACH_SD, 0x8000_4f10, 0x0140_0008); + assert_eq!(c.cache_op(C_R10K_ILD | CACH_PI, set(0x10), 0), 0x3_4420_0004, "a held primary line answers for itself"); + c.cache_op(C_IST | CACH_PI, set(0x10), 0); + c.cache_op(C_IST | CACH_SD, 0x8000_4f00, 0); + assert_eq!(c.cache_op(C_R10K_ILD | CACH_PI, set(0x10), 0), 0x3_4420_0004, "gone with the line: the array"); + } + + /// A page the cache's bus counts on a counter of its own (a ROM mapped + /// over memory) carries one `GEN_HELD` there however many lines hold it, + /// and none once they go. A counter that is ppmem's is left to ppmem. + #[cfg(feature = "jitv2")] + #[test] + fn a_held_page_marks_the_buss_own_counter() { + use crate::cpu::jitv2::jitv2::GEN_HELD; + use std::sync::atomic::{AtomicU64, Ordering}; + struct Rom(AtomicU64); + impl BusDevice for Rom { + fn gen_ptr(&self, _addr: u64) -> *const AtomicU64 { &self.0 } + } + let (space, _banks) = PpMemSpace::with_bank_sizes(&[64]).unwrap(); + let space = Box::new(space); + space.map_bank(0, 0, 64 << 20, 64 << 20).unwrap(); + let rom = Arc::new(Rom(AtomicU64::new(0))); + let c = Box::new(R10000ShadowCache::from(rom.clone() as Arc)); + attach(&c, &space); + let marks = || rom.0.load(Ordering::Relaxed) / GEN_HELD; + c.cache_op(C_IST | CACH_PD, 0x8000_0000, pd_dirty(0x2000)); + c.cache_op(C_IST | CACH_PD, 0x8000_0040, pd_dirty(0x2040)); + assert_eq!(marks(), 1, "one mark for two lines"); + c.cache_op(C_IST | CACH_PD, 0x8000_0000, 0); + assert_eq!(marks(), 1, "the page is still held"); + c.cache_op(C_IST | CACH_PD, 0x8000_0040, 0); + assert_eq!(marks(), 0, "off with the last line"); + assert_eq!(rom.0.load(Ordering::Relaxed), 1, "by one ordinary bump"); + } } diff --git a/src/cpu/mips_cache_v2.rs b/src/cpu/mips_cache_v2.rs index 2b92dc849..5619a9cb3 100644 --- a/src/cpu/mips_cache_v2.rs +++ b/src/cpu/mips_cache_v2.rs @@ -669,12 +669,6 @@ pub trait MipsCache: Send + Sync { /// Share the L1-I hit/fetch counters with the status display. No-op where absent. fn set_l1i_counters(&mut self, _hit: Arc, _fetch: Arc) {} - /// Have `hook` called with TagLo on every Index Store Tag to the primary - /// data cache: a machine whose firmware builds memory out of cache tags - /// (the O2's POST stack, see `O2Memory::stack_tag_store`). No-op where - /// absent. Set once, before the CPU runs. - fn set_pd_tag_store_hook(&self, _hook: Box) {} - /// tcache: point the cache at ppmem's window and mapped-region bitmap, so /// cacheable RAM accesses read/write RAM directly instead of copying line /// data (docs/tcache-design.md). Default no-op — a cache with no @@ -684,6 +678,15 @@ pub trait MipsCache: Send + Sync { /// `base` must be ppmem's window base and `bitmap` a live `u64` that /// outlives this cache. unsafe fn set_tcache_window(&self, _base: *mut u8) {} + + /// The cached view this cache reads memory through + /// (`PpMemSpace::add_cached_view`), as `sp` and its number, so valid + /// lines can hold their pages there. Set with the window, before the CPU + /// runs. No-op where the cache holds no lines. + /// + /// # Safety + /// `sp` must outlive the cache. + unsafe fn set_cached_view(&self, _sp: *const crate::ppmem::PpMemSpace, _view: usize) {} /// The ppmem window's `window_shift`, when it is not the 4GB window's. fn set_tcache_shift(&self, _shift: u32) {} diff --git a/src/cpu/mips_exec.rs b/src/cpu/mips_exec.rs index 782883bf7..1e97f1b60 100644 --- a/src/cpu/mips_exec.rs +++ b/src/cpu/mips_exec.rs @@ -998,6 +998,14 @@ pub struct LlSiteStats { pub struct MipsExecutor { pub core: MipsCore, pub sysad: Arc, + /// Where the JIT reads code and generation counters, and the bus its + /// compile pool is started on: cacheable memory at plain physical + /// addresses. The same object as `sysad` unless the board's uncached + /// connection decodes a plain address as something other than memory (the + /// Fuel's decodes it as Hub-special space, with RAM only in its first + /// 256 MiB), in which case [`Self::set_cached_bus`] gives it the view the + /// cache fills from. + cached_bus: Arc, /// cheritest convention (`--cheritest-dump-hook`): a guest write to CP0 26 /// dumps machine state via the test device. Off unless asked for — CP0 26 /// is ECC on a real R4400 and the PROM writes it during cache init. @@ -2541,6 +2549,17 @@ impl MipsExecutor { size * 8, self.core.pc, virt_addr, phys_addr, val, status, self.fault_note(phys_addr, size)); } + /// The bus the JIT reads code from and its compile pool is started on. + pub fn cached_bus(&self) -> &Arc { + &self.cached_bus + } + + /// Give the JIT a cacheable view of memory that differs from `sysad` (see + /// the field). A board that does so builds the cache on the same view. + pub fn set_cached_bus(&mut self, bus: Arc) { + self.cached_bus = bus; + } + /// Create a new executor from a config and a bus (sysad) and a TLB. /// The cache hierarchy is constructed internally as a unified R4000Cache. pub fn new(sysad: Arc, mut tlb: T, cfg: &MipsCpuConfig) -> Self @@ -2696,6 +2715,7 @@ impl MipsExecutor { let mut executor = Self { core, + cached_bus: sysad.clone(), sysad, cheritest_dump_hook: false, tlb, @@ -2872,6 +2892,14 @@ impl MipsExecutor { unsafe { self.cache.set_tcache_window(base) }; } + /// See `MipsCache::set_cached_view`. + /// + /// # Safety + /// `sp` must outlive the executor. + pub unsafe fn set_cached_view(&self, sp: *const crate::ppmem::PpMemSpace, view: usize) { + unsafe { self.cache.set_cached_view(sp, view) }; + } + /// tcache: the window's `ppmem::window_shift`, when it is not 4GB's. pub fn set_tcache_shift(&self, shift: u32) { self.cache.set_tcache_shift(shift); @@ -3382,7 +3410,7 @@ va={:#018x} phys={:#010x} (code pfn {:#x}, page {:#010x}, word {}/{})", #[cfg(feature = "jitv2")] pub(crate) fn jitv2_flush_from_here(&mut self, only_if_exhausted: bool) { crate::pf_scope!(crate::perfetto::events::JIT_CPU_FLUSH); - let bus = self.sysad.clone(); + let bus = self.cached_bus.clone(); let jit = self.jitv2.clone(); let flush = || { let mut j = jit.lock(); @@ -3524,7 +3552,7 @@ va={:#018x} phys={:#010x} (code pfn {:#x}, page {:#010x}, word {}/{})", let (mut jit, lookup) = { crate::pf_scope!(crate::perfetto::events::JIT_CPU_TRACK); let mut jit = self.jitv2.lock(); - let lookup = jit.page_for(pfn, page_base, self.sysad.as_ref(), fr1); + let lookup = jit.page_for(pfn, page_base, self.cached_bus.as_ref(), fr1); (jit, lookup) }; match lookup { @@ -3565,7 +3593,7 @@ va={:#018x} phys={:#010x} (code pfn {:#x}, page {:#010x}, word {}/{})", // exact page this fetch already landed on. self.clear_pcp(); let mut jit = self.jitv2.lock(); - let slot = jit.page_for(pfn, page_base, self.sysad.as_ref(), fr1) + let slot = jit.page_for(pfn, page_base, self.cached_bus.as_ref(), fr1) .expect("flush must leave room for at least one page"); // Direct assignment for the same borrow reason as the // hit path above; `pfn` is what `set_pcp` would mirror. @@ -9317,7 +9345,7 @@ va={:#018x} phys={:#010x} (code pfn {:#x}, page {:#010x}, word {}/{})", .is_some_and(|c| c.packing_stats().1 > crate::cpu::jitv2::CODEGEN_ARENA_FLUSH_THRESHOLD_BYTES); if over_threshold { let phys_page_base = page.pfn as u64 * crate::cpu::jitv2::PAGE_SIZE as u64; - unsafe { self.jitv2.lock().flush_from_cpu_thread(self.sysad.clone()); } + unsafe { self.jitv2.lock().flush_from_cpu_thread(self.cached_bus.clone()); } self.clear_pcp(); self.jitv2_track_pcp(phys_page_base); return EXEC_RETRY; @@ -9339,10 +9367,10 @@ va={:#018x} phys={:#010x} (code pfn {:#x}, page {:#010x}, word {}/{})", #[cfg(feature = "developer")] { let stats = self.jitv2.lock().stats.clone(); - ran_out_of_memory = crate::cpu::jitv2::comp::handle_request(&req, &self.sysad, &mut self.jitv2_inline_analyzer, codegen, &stats); + ran_out_of_memory = crate::cpu::jitv2::comp::handle_request(&req, &self.cached_bus, &mut self.jitv2_inline_analyzer, codegen, &stats); } #[cfg(not(feature = "developer"))] - { ran_out_of_memory = crate::cpu::jitv2::comp::handle_request(&req, &self.sysad, &mut self.jitv2_inline_analyzer, codegen); } + { ran_out_of_memory = crate::cpu::jitv2::comp::handle_request(&req, &self.cached_bus, &mut self.jitv2_inline_analyzer, codegen); } } *self.jitv2.lock().codegen.lock() = codegen; { @@ -9353,7 +9381,7 @@ va={:#018x} phys={:#010x} (code pfn {:#x}, page {:#010x}, word {}/{})", if ran_out_of_memory { let page = unsafe { &mut *self.pcp }; let phys_page_base = page.pfn as u64 * crate::cpu::jitv2::PAGE_SIZE as u64; - unsafe { self.jitv2.lock().flush_from_cpu_thread(self.sysad.clone()); } + unsafe { self.jitv2.lock().flush_from_cpu_thread(self.cached_bus.clone()); } self.clear_pcp(); self.jitv2_track_pcp(phys_page_base); return EXEC_RETRY; @@ -10561,7 +10589,7 @@ impl MipsCpu { } /// ppmem: pointer to the executor's inline `ppmem_bitmap` field, for - /// `PpMemSpace::set_bitmap_sink`. Fixed for the life of this `MipsCpu` — + /// `PpMemSpace::add_cached_view`. Fixed for the life of this `MipsCpu` — /// taken once in `new()` after the executor reached its final address. pub fn ppmem_bitmap_ptr(&self) -> *mut u64 { self.ppmem_bitmap_ptr @@ -10575,6 +10603,14 @@ impl MipsCpu { unsafe { self.executor.lock().set_tcache_window(base) }; } + /// See `MipsExecutor::set_cached_view`. + /// + /// # Safety + /// `sp` must outlive the CPU. + pub unsafe fn set_cached_view(&self, sp: *const crate::ppmem::PpMemSpace, view: usize) { + unsafe { self.executor.lock().set_cached_view(sp, view) }; + } + /// tcache: see `MipsExecutor::set_tcache_shift`. pub fn set_tcache_shift(&self, shift: u32) { self.executor.lock().set_tcache_shift(shift); @@ -11293,13 +11329,18 @@ pub trait CpuDevice: Device + Resettable + Saveable + Send + Sync { fn cycles_ptr(&self) -> crate::cpu::mips_core::CyclesPtr; fn interrupts_ptr(&self) -> *const AtomicU64; /// ppmem: the executor's inline `ppmem_bitmap` field, for - /// `PpMemSpace::set_bitmap_sink`. + /// `PpMemSpace::add_cached_view`. fn ppmem_bitmap_ptr(&self) -> *mut u64; /// tcache: hand the cache ppmem's window base + mapped-region bitmap. /// /// # Safety /// Same contract as `CpuCache::set_tcache_window`. unsafe fn set_tcache_window(&self, base: *mut u8); + /// See `MipsCache::set_cached_view`. + /// + /// # Safety + /// `sp` must outlive the CPU. + unsafe fn set_cached_view(&self, sp: *const crate::ppmem::PpMemSpace, view: usize); /// tcache: see `MipsExecutor::set_tcache_shift`. fn set_tcache_shift(&self, shift: u32); /// tcache: pointer to the cache's inline mapped-region bitmap. @@ -11367,6 +11408,9 @@ impl CpuDevice for MipsCp unsafe fn set_tcache_window(&self, base: *mut u8) { unsafe { MipsCpu::set_tcache_window(self, base) } } + unsafe fn set_cached_view(&self, sp: *const crate::ppmem::PpMemSpace, view: usize) { + unsafe { MipsCpu::set_cached_view(self, sp, view) } + } fn tcache_bitmap_ptr(&self) -> *mut u64 { MipsCpu::tcache_bitmap_ptr(self) } #[cfg(feature = "jitv2")] unsafe fn set_tcache_gen_window(&self, gen_base: *mut AtomicU64) { diff --git a/src/cpu/mips_exec/monitor.rs b/src/cpu/mips_exec/monitor.rs index a4385fad1..c836309ed 100644 --- a/src/cpu/mips_exec/monitor.rs +++ b/src/cpu/mips_exec/monitor.rs @@ -2027,7 +2027,7 @@ impl MipsCpu { #[cfg(not(feature = "jitv2_lockstep"))] { exec.jitv2_inline_compile = false; - let bus = exec.sysad.clone(); + let bus = exec.cached_bus.clone(); let old = exec.jitv2.lock().codegen.lock().take(); match old { Some(old) => { @@ -3010,7 +3010,7 @@ impl MipsCpu { // implementation. let path = actual_args.get(1).copied().unwrap_or("jitv2.html"); let jitv2_arc = exec.jitv2.clone(); - let bus = exec.sysad.clone(); + let bus = exec.cached_bus.clone(); crate::cpu::jitv2_html_j2wp::write_jitv2_html( &jitv2_arc, &bus, &exec.memory.clone(), &mut exec.jitv2_inline_analyzer, path, &mut *writer, )?; diff --git a/src/cpu/mips_exec_test.rs b/src/cpu/mips_exec_test.rs index 4127eff99..0714c8d88 100644 --- a/src/cpu/mips_exec_test.rs +++ b/src/cpu/mips_exec_test.rs @@ -4282,6 +4282,62 @@ mod tests { check::(0xF_FFFF_FFFF); } + /// A mapped uncached page's attribute (EntryLo 63:62, carried to physical + /// bits 58:57) reaches the bus on a board that decodes them, and is folded + /// away on the Octane (40 bits), Indigo2 and O2 (32), which is what keeps + /// those machines unchanged. + #[test] + fn mapped_uncached_attribute_is_folded_by_the_boards_physical_mask() { + std::thread::Builder::new() + .stack_size(16 * 1024 * 1024) + .spawn(|| { + use crate::cpu::mips_tlb::MipsTlb; + const VA: u64 = 0x4000_0000; + + fn translate>>( + phys_mask: u64, alias: Option<(u64, u64, u64)>, pfn: u64, + ) -> u64 { + let bus: Arc = Arc::new(MockMemory::new()); + let cfg = MipsCpuConfig::for_model::(Arc::new(crate::ip22_bus::memory_map(false))); + let mut exec = Box::new(MipsExecutor::::new( + bus, MipsTlb::default(), &cfg)); + disable_jitv2(&mut exec); + exec.install_status_cb(); + let dmtc0 = |exec: &mut Box>, rd: u32, value: u64| { + exec.core.write_gpr(1, value); + let op = (OP_COP0 << 26) | (RS_DMTC0 << 21) | (1 << 16) | (rd << 11); + assert_eq!(exec.exec(op), EXEC_COMPLETE); + }; + let status = (exec.core.cp0_status & !crate::cpu::mips_core::STATUS_ERL) | STATUS_KX; + dmtc0(&mut exec, 12, status as u64); + dmtc0(&mut exec, 10, VA); + dmtc0(&mut exec, 0, 0); + dmtc0(&mut exec, 5, 0); + // C=2 (uncached), valid, dirty, attribute 1 in bits 63:62. + dmtc0(&mut exec, 2, (1 << 62) | (pfn << 6) | (2 << 3) | (1 << 2) | (1 << 1)); + dmtc0(&mut exec, 3, 0); + assert_eq!(exec.exec((OP_COP0 << 26) | (RS_TLB << 21) | FUNCT_TLBWI), EXEC_COMPLETE); + exec.phys_mask = phys_mask; + exec.phys_alias = alias; + exec.debug_translate(VA).phys + } + + type R10K = crate::cpu::mips_cache_shadow::R10000ShadowCache; + type R14K = crate::cpu::mips_cache_shadow::R12000ShadowCache; + const PA: u64 = 0x0d10_0000; + assert_eq!(translate::(u64::MAX, None, PA >> 12), (1 << 57) | PA, + "no mask: the attribute reaches the bus"); + for (name, mask) in [("Octane", (1u64 << 40) - 1), ("Indigo2/O2", u32::MAX as u64)] { + assert_eq!(translate::(mask, None, PA >> 12), PA, "{name} folds it away"); + } + assert_eq!(translate::(u64::MAX, None, PA >> 12), PA, + "a model without the attribute never stores it"); + }) + .unwrap() + .join() + .unwrap(); + } + /// The Indy, Indigo2 and O2 decode 32 physical bits, and IRIX relies on /// it: on the IP28 the kernel stack's wired entry carries PTE software /// bits as PA[34:33]. `phys_mask` must fold them away before the bus. diff --git a/src/cpu/mips_tlb.rs b/src/cpu/mips_tlb.rs index 3eaed8e8c..ff504582f 100644 --- a/src/cpu/mips_tlb.rs +++ b/src/cpu/mips_tlb.rs @@ -147,6 +147,23 @@ pub enum AccessType { Debug = 3, } +/// The physical base a TLB entry's EntryLo maps. A model that keeps the +/// R10000 family's uncached attribute (EntryLo bits 63:62, see +/// `CpuModel::R10K_FAMILY`) has it carried to physical bits 58:57 +/// for an uncached page, where an unmapped xkphys access carries it; other +/// models never store those bits. For a mapped reference the R10000 takes the +/// two-bit uncached attribute from the TLB's uncached-attribute field, loaded +/// from EntryLo0/1 bits 63:62 (NEC U10278EJ4V0UM, VR10000/VR12000 user's +/// manual, 4th edition, section 11.3; R10000 User's Manual 007-2490-001). +/// +/// Whether the page is uncached is the caller's decode of C through the model's +/// table (`MipsTlb::c_attr`), not another decode here. +#[inline] +fn pfn_base(lo: u64, pfn_mask: u64, uncached: bool) -> u64 { + let attribute = (((lo >> 62) & 3) << 57) & 0u64.wrapping_sub(uncached as u64); + ((lo << 6) & pfn_mask) | attribute +} + /// TLB Entry structure matching R4000 TLB format #[derive(Debug, Clone, Copy, PartialEq, Eq)] pub struct TlbEntry { @@ -1008,6 +1025,21 @@ impl Default for MipsTlb { } } +impl MipsTlb { + /// `pfn_base` for both halves of an entry, deciding "uncached" with the + /// model's C table. + #[inline] + fn pfn_bases(&self, lo: &[u64; 2], pfn_mask: u64) -> [u64; 2] { + // No attribute bits set, the usual case: nothing to carry, and the C + // table need not be read. + if (lo[0] | lo[1]) >> 62 == 0 { + return [(lo[0] << 6) & pfn_mask, (lo[1] << 6) & pfn_mask]; + } + let uncached = |lo: u64| self.c_attr[((lo >> 3) & 7) as usize] == CacheAttr::Uncached; + [pfn_base(lo[0], pfn_mask, uncached(lo[0])), pfn_base(lo[1], pfn_mask, uncached(lo[1]))] + } +} + impl Tlb for MipsTlb { #[inline] fn translate(&mut self, virt_addr: u64, asid: u8, access_type: AccessType) -> TlbResult { @@ -1177,10 +1209,7 @@ impl Tlb for MipsTlb { entry.vpn_hi64 = entry.entry_hi & entry.vcmp64; entry.offset_mask = (1u64 << entry.selector_bit_shift) - 1; let pfn_mask = 0xFF_FFFF_FFFF_000 & !entry.offset_mask; - entry.pfn_base = [ - (entry.entry_lo[0] << 6) & pfn_mask, - (entry.entry_lo[1] << 6) & pfn_mask, - ]; + entry.pfn_base = self.pfn_bases(&entry.entry_lo, pfn_mask); self.irregular_mask |= mask & (mask + 1) != 0; if self.invalidate_on_write { self.invalidate_overlapped(index, &entry); @@ -1322,6 +1351,10 @@ impl Tlb for MipsTlb { fn set_cache_attrs(&mut self, table: [CacheAttr; 8]) { self.c_attr = table; for i in 0..self.num_entries { + // pfn_base carries the attribute for pages the new table decodes + // as uncached, so it follows the table. + let pfn_mask = 0xFF_FFFF_FFFF_000 & !self.entries[i].offset_mask; + self.entries[i].pfn_base = self.pfn_bases(&self.entries[i].entry_lo, pfn_mask); self.shadow[i] = ShadowEntry::from_entry(&self.entries[i], &self.c_attr); } } @@ -1380,7 +1413,7 @@ impl Tlb for MipsTlb { vcmp64, vpn_hi64: entry_hi & vcmp64, offset_mask, - pfn_base: [(lo[0] << 6) & pfn_mask, (lo[1] << 6) & pfn_mask], + pfn_base: self.pfn_bases(&lo, pfn_mask), }; } } diff --git a/src/cpu/mips_tlb_test.rs b/src/cpu/mips_tlb_test.rs index be121f291..65ca06067 100644 --- a/src/cpu/mips_tlb_test.rs +++ b/src/cpu/mips_tlb_test.rs @@ -535,3 +535,52 @@ fn tlb_write_inside_a_larger_entry_count_is_kept() { assert_eq!(r10000.read(63).entry_hi, entry.entry_hi, "...and its last slot"); } } + +/// A mapped uncached page carries EntryLo's uncached attribute (bits 63:62) +/// to the bus as physical bits 58:57; a cached page never does, whatever those bits +/// hold, and no attribute leaves the PFN alone. "Uncached" is the model's C +/// table, so a model that decodes a C value differently, or a table installed +/// after the entry was written, changes the answer. +#[test] +fn only_uncached_pages_carry_the_uncached_attribute() { + use super::{MipsTlb, Tlb, TlbEntry}; + use crate::cpu::mips_exec::CacheAttr; + // MipsTlb is large; give the one instance room, as its neighbours do. + std::thread::Builder::new() + .stack_size(16 * 1024 * 1024) + .spawn(|| { + let pfn = 0x0d100u64 << 6; + let base = |tlb: &MipsTlb| tlb.read(0).pfn_base; + let write = |tlb: &mut MipsTlb, lo0: u64, lo1: u64| { + let mut entry = TlbEntry::new(); + entry.entry_hi = 0xC000_0000_0004_0000; + entry.entry_lo = [lo0, lo1]; + tlb.write(0, entry); + }; + let mut tlb = Box::new(MipsTlb::new(64)); + // The default table: C=2 and the reserved values are uncached. + for (cca, expect) in [(2u64, 1u64 << 57), (6, 1 << 57), (3, 0), (5, 0)] { + let lo = (1 << 62) | pfn | (cca << 3) | 0x6; + write(&mut tlb, lo, lo); + assert_eq!(base(&tlb), [0x0d10_0000 | expect; 2], "cca {cca}"); + } + write(&mut tlb, pfn | (2 << 3), 0); + assert_eq!(base(&tlb)[0], 0x0d10_0000, "no attribute leaves the PFN alone"); + + // The R10000 table reads C=7 (uncached accelerated) as uncached and + // 5 as cacheable; a model that read 7 as cacheable carries nothing. + let mut table = [CacheAttr::Uncached; 8]; + table[3] = CacheAttr::Cacheable; + table[5] = CacheAttr::CacheableCoherent; + tlb.set_cache_attrs(table); + let lo = (1 << 62) | pfn | (7 << 3) | 0x6; + write(&mut tlb, lo, lo); + assert_eq!(base(&tlb), [0x0d10_0000 | 1 << 57; 2], "C=7 uncached under this table"); + table[7] = CacheAttr::Cacheable; + tlb.set_cache_attrs(table); + assert_eq!(base(&tlb), [0x0d10_0000; 2], "the table, installed later, decides"); + }) + .unwrap() + .join() + .unwrap(); +} diff --git a/src/cpu/parts.rs b/src/cpu/parts.rs index ceda3972c..b3d0479e3 100644 --- a/src/cpu/parts.rs +++ b/src/cpu/parts.rs @@ -52,6 +52,9 @@ pub const R10000_250: Part = Part { mhz: 250, rev: (3, 4), ..R10000_195 }; pub const R12000_360: Part = Part { family: Family::R12000, mhz: 360, rev: (3, 5), l2_kb: 2048 }; pub const R12000_400: Part = Part { mhz: 400, ..R12000_360 }; pub const R14000_600: Part = Part { family: Family::R14000, mhz: 600, rev: (2, 4), l2_kb: 2048 }; +/// The Fuel's R14000A: rev 2.3 at 500 MHz with a 2 MB secondary cache, as the IP35 +/// `hinv` dumps show it. +pub const R14000_500: Part = Part { mhz: 500, rev: (2, 3), ..R14000_600 }; /// Every part, for the bare-metal bench rig, which takes any. A name matches /// the first listed, so the rig runs the later silicon of a clock that has two. diff --git a/src/dev/aic7880.rs b/src/dev/aic7880.rs index 5c828a2e3..78fcad99f 100644 --- a/src/dev/aic7880.rs +++ b/src/dev/aic7880.rs @@ -420,11 +420,12 @@ pub struct Aic7880 { /// CHDs, overlays, CD-ROMs). An install needs at least two -- the media /// to read and the disk to write. bus: ScsiBus, - /// Host memory, through the PCI bridge the chip masters. The controller - /// fetches its work from there rather than being handed it a register at - /// a time, so without this it can see the tag of a command and nothing - /// else about it. - dma: std::sync::OnceLock>, + /// The configuration header, the interrupt pin as the board routes it, + /// and host memory through the PCI bridge the chip masters. The + /// controller fetches its work from host memory rather than being handed + /// it a register at a time, so without it the chip sees the tag of a + /// command and nothing else about it. + pci: crate::pci::PciFunctionState, /// The SCSI id a disk attached without one answers at. target: u8, /// The status bar's activity bits, shared with the display thread. @@ -439,9 +440,6 @@ pub struct Aic7880 { sense: Mutex>, /// Pages of DMA addresses already reported as not memory. dma_misses: Mutex>, - /// The chip's interrupt pin, as the board routes it. - irq: std::sync::OnceLock, - pci: crate::pci::PciConfig, } /// The chip's PCI identity: Adaptec, AIC-7880. @@ -476,16 +474,14 @@ impl Aic7880 { running: AtomicBool::new(false), me: me.clone(), bus, - dma: std::sync::OnceLock::new(), sense: Mutex::new(None), dma_misses: Mutex::new(std::collections::BTreeSet::new()), - irq: std::sync::OnceLock::new(), // 256 bytes of I/O and 4 KiB of memory, both onto the same // registers: what the part has. pci: { - use crate::pci::{Bar, PciConfig, PCI_INTA}; - PciConfig::new(PCI_VENDOR_ADAPTEC, PCI_DEVICE_AIC7880, PCI_CLASS_SCSI, 0x01, - [Bar::Io(0x100), Bar::Mem(0x1000), Bar::Unused, Bar::Unused, Bar::Unused, Bar::Unused], PCI_INTA) + use crate::pci::{Bar, PciConfig, PciFunctionState, PCI_INTA}; + PciFunctionState::new(PciConfig::new(PCI_VENDOR_ADAPTEC, PCI_DEVICE_AIC7880, PCI_CLASS_SCSI, 0x01, + [Bar::Io(0x100), Bar::Mem(0x1000), Bar::Unused, Bar::Unused, Bar::Unused, Bar::Unused], PCI_INTA)) }, target: 1, heartbeat: std::sync::OnceLock::new(), @@ -503,7 +499,7 @@ impl Aic7880 { /// CDB pointer inside it is `0x0107532c`. fn dma_read_bytes(&self, addr: u32, n: usize) -> Vec { let mut b = vec![0u8; n]; - if !self.dma.get().is_some_and(|d| d.dma_read(addr as u64, &mut b).is_ok()) { + if !self.pci.dma().is_some_and(|d| d.dma_read(addr as u64, &mut b).is_ok()) { self.dma_miss(addr, false); } b @@ -514,7 +510,7 @@ impl Aic7880 { } fn dma_write_bytes(&self, addr: u32, data: &[u8]) { - if !self.dma.get().is_some_and(|d| d.dma_write(addr as u64, data).is_ok()) { + if !self.pci.dma().is_some_and(|d| d.dma_write(addr as u64, data).is_ok()) { self.dma_miss(addr, true); } } @@ -585,12 +581,6 @@ impl Aic7880 { self.st.lock().intstat & 0x0f != 0 } - /// Wire the chip's interrupt pin to `line`. - pub fn connect_irq(&self, line: crate::irq::IrqLine) { - let _ = self.irq.set(line); - self.update_irq(); - } - /// Drive the interrupt pin to the chip's state. fn update_irq(&self) { self.drive_irq(&self.st.lock()); @@ -601,7 +591,7 @@ impl Aic7880 { /// same lock as the change; set after releasing it, a completion's /// "high" could land after the host's clear and stick. fn drive_irq(&self, st: &State) { - if let Some(l) = self.irq.get() { + if let Some(l) = self.pci.irq() { l.set(st.intstat & 0x0f != 0); } } @@ -665,7 +655,7 @@ impl Aic7880 { /// bytes the HIM also uses, so a doorbell whose rings are not memory is /// somebody else's bytes and not a doorbell. fn is_memory(&self, addr: u32) -> bool { - self.dma.get().is_some_and(|d| d.dma_read(addr as u64, &mut [0u8]).is_ok()) + self.pci.dma().is_some_and(|d| d.dma_read(addr as u64, &mut [0u8]).is_ok()) } /// A CDB's length follows from its opcode group. Taking it from the @@ -1691,14 +1681,19 @@ impl crate::traits::Saveable for Aic7880 { Some(a) if a.len() == 3 => Some((toml_u8(&a[0]).unwrap_or(0), toml_u8(&a[1]).unwrap_or(0), toml_u8(&a[2]).unwrap_or(0))), _ => None, }; + // The snapshot holds the chip, not the SCSI bus behind it. A fresh + // process has its targets as built, and this one may have a CD-ROM + // at 512 bytes from the last driver, so put them where a bus reset + // leaves them under the restored driver (see `reset_targets`). + self.reset_targets(); self.update_irq(); Ok(()) } } /// Both BARs reach the same registers. -impl crate::pci::PciBusDevice for Aic7880 { - fn config(&self) -> &crate::pci::PciConfig { +impl crate::pci::PciFunction for Aic7880 { + fn state(&self) -> &crate::pci::PciFunctionState { &self.pci } @@ -1710,12 +1705,9 @@ impl crate::pci::PciBusDevice for Aic7880 { self.write(off, val, width) } - fn connect_interrupt(&self, line: crate::irq::IrqLine) { - self.connect_irq(line); - } - - fn connect_dma(&self, dma: std::sync::Arc) { - let _ = self.dma.set(dma); + /// Drive the pin the bus wired to the chip's state. + fn attached(&self) { + self.update_irq(); } } @@ -1760,7 +1752,7 @@ mod tests { /// A chip that masters host memory `ram` through the bridge. fn on(ram: Ram) -> Arc { let c = Aic7880::new(); - crate::pci::PciBusDevice::connect_dma(&*c, std::sync::Arc::new(crate::dev::mace_pci::HostMemory::new(ram))); + crate::pci::PciFunction::state(&*c).connect_dma(std::sync::Arc::new(crate::dev::mace_pci::HostMemory::new(ram))); c } @@ -2307,13 +2299,50 @@ mod tests { assert_eq!(c.read(reg::QINCNT, 1), 0); } + /// A restore into a chip that has been in use equals one into a fresh + /// chip, targets included: the snapshot holds the chip, not the SCSI + /// bus, so the targets go back to what a bus reset leaves under the + /// restored driver (a CD-ROM at 512 bytes under SGI's), whatever the + /// controller had done to them before. + #[test] + fn aic7880_load_into_a_dirtied_instance_equals_load_into_a_fresh_one() { + use crate::traits::Saveable; + let make = |name: &str| { + let (c, ram) = him_rig(name); + let cd = image_path(&format!("{name}-cd")); + std::fs::write(&cd, vec![0u8; 64 * BLOCK_SIZE]).unwrap(); + c.bus().attach_image(3, &cd, true).expect("attach cd"); + (c, ram) + }; + let block_sizes = |c: &Aic7880| { + let mut sizes = Vec::new(); + c.bus().for_each(|id, d| sizes.push((id, d.block_size()))); + sizes + }; + let (src, ram) = make("ld-src"); + put_cmd(&ram, 2, 1, &TUR, 0); + src.write(reg::QINFIFO, 2, 1); + let v = src.save_state(); + + let (used, _) = make("ld-used"); + used.bus().for_each(|_, d| d.reset_block_size(true)); + let (fresh, _) = make("ld-fresh"); + assert_ne!(block_sizes(&used), block_sizes(&fresh), "the dirtied bus differs before the load"); + used.load_state(&v).unwrap(); + fresh.load_state(&v).unwrap(); + assert_eq!(used.save_state(), fresh.save_state()); + assert_eq!(block_sizes(&used), block_sizes(&fresh), "and agrees after"); + assert!(block_sizes(&used).contains(&(3, Some(512))), "a CD-ROM is at 512 under the HIM"); + } + /// Nobody at an id: the selection times out, the way the HIM expects, /// and nothing goes to the queue-out. #[test] fn an_empty_id_times_out_its_selection() { let (c, ram) = him_rig("him-selto"); let (irq, line) = crate::irq::LevelProbe::line(); - c.connect_irq(line); + crate::pci::PciFunction::state(&*c).connect_irq(line); + crate::pci::PciFunction::attached(&*c); let inquiry = [0x12, 0, 0, 0, 36, 0]; put_cmd(&ram, 2, 3, &inquiry, 36); c.write(reg::QINFIFO, 2, 1); @@ -2476,7 +2505,8 @@ mod tests { use crate::traits::Device; let (c, ram) = him_rig("worker"); let irq = std::sync::Arc::new(crate::irq::LevelProbe::default()); - c.connect_irq(crate::irq::IrqLine::new(irq.clone(), 0)); + crate::pci::PciFunction::state(&*c).connect_irq(crate::irq::IrqLine::new(irq.clone(), 0)); + crate::pci::PciFunction::attached(&*c); put_cmd(&ram, 0, 1, &TUR, 0); put_cmd(&ram, 1, 1, &TUR, 0); c.start(); diff --git a/src/dev/bedrock_flash.rs b/src/dev/bedrock_flash.rs new file mode 100644 index 000000000..7df30a0cc --- /dev/null +++ b/src/dev/bedrock_flash.rs @@ -0,0 +1,638 @@ +//! Bedrock's boot flash: two independent 8 MiB arrays, selected 32 MiB +//! apart, with the AMD command set and CFI query data. +//! +//! Boot aliases and executable views share the same array bytes. Each array +//! has its own command state; instruction fetches read stored bytes even +//! while register reads return command status. +//! +//! Ported from hw/misc/sgi-bedrock-flash.c and hw/mips/sgi/initial-flash.c +//! in jamesbraid/qemu-sgi-origami. + + + +use super::sflash::amd::{AmdCore, Effect, SECTOR_SIZE}; +use super::sflash::storage::FlashBacking; +use crate::traits::{ + BusDevice, BusRead16, BusRead32, BusRead64, BusRead8, Resettable, BUS_ERR, BUS_OK, +}; +use parking_lot::Mutex; +use std::io; +use std::path::Path; +use std::sync::{ + atomic::{AtomicU64, Ordering}, + Arc, +}; + +pub const PART_SIZE: usize = 8 << 20; +pub const FLASH_SIZE: usize = 2 * PART_SIZE; +pub const SECOND_POSITION: u64 = 32 << 20; + +fn sum(bytes: &[u8]) -> u8 { + bytes.iter().fold(0u8, |s, b| s.wrapping_add(*b)) +} + +/// The configuration block's CPU type word: 0 for the R10000 and R12000, 1 +/// for the R14000, 2 for the R16000, as the IP35 PROM's own configuration +/// editor lists them. The Fuel has an R14000. +const CPU_TYPE_R14000: u32 = 1; + +/// Apply Fuel's firmware record fields while retaining the configuration sum. +/// This policy also applies when reopening an existing persistent store. +fn configure(data: &mut [u8]) { + let config = &mut data[0x60..0xb0]; + let old = sum(config); + config[0x34..0x38].fill(0); + config[4..8].copy_from_slice(&0x549aa985u32.to_be_bytes()); + config[0x14..0x18].copy_from_slice(&500_000_000u32.to_be_bytes()); + config[0x30..0x34].copy_from_slice(&CPU_TYPE_R14000.to_be_bytes()); + config[0x38..0x3c].copy_from_slice(&1u32.to_be_bytes()); + let adjustment = old.wrapping_sub(sum(config)); + config[0x34..0x38].copy_from_slice(&(adjustment as u32).to_be_bytes()); +} + +struct State { + data: Vec, + parts: [AmdCore; 2], + backing: Option, +} + +impl State { + fn new(data: Vec) -> Self { + Self { + data, + parts: std::array::from_fn(|_| AmdCore::default()), + backing: None, + } + } + // Keep the in-memory array unchanged until each backing write succeeds. + // If a later range fails, restore the old bytes for all ranges already tried. + fn persist(&mut self, ranges: &[(usize, &[u8])]) -> io::Result<()> { + let Some(backing) = &mut self.backing else { + return Ok(()); + }; + for (index, &(offset, bytes)) in ranges.iter().enumerate() { + if let Err(error) = backing.write_range(offset, bytes) { + for &(restore_offset, restore_bytes) in &ranges[..=index] { + let old = &self.data[restore_offset..restore_offset + restore_bytes.len()]; + if let Err(rollback) = backing.write_range(restore_offset, old) { + log::error!( + "bedrock-flash: backing rollback at {restore_offset:#x} failed: {rollback}" + ); + } + } + return Err(error); + } + } + Ok(()) + } + fn replace(&mut self, offset: usize, bytes: &[u8]) -> io::Result { + if self.data[offset..offset + bytes.len()] == *bytes { + return Ok(false); + } + self.persist(&[(offset, bytes)])?; + self.data[offset..offset + bytes.len()].copy_from_slice(bytes); + Ok(true) + } + fn erase(&mut self, part: usize, sectors: &[usize]) -> io::Result { + let erased = vec![0xff; SECTOR_SIZE]; + let ranges: Vec<_> = sectors + .iter() + .map(|sector| (part * PART_SIZE + sector * SECTOR_SIZE, erased.as_slice())) + .collect(); + self.persist(&ranges)?; + let mut changed = false; + for (offset, bytes) in ranges { + let data = &mut self.data[offset..offset + bytes.len()]; + changed |= data.iter().any(|byte| *byte != 0xff); + data.fill(0xff); + } + Ok(changed) + } +} + +pub struct Flash { + st: Mutex, + gen: AtomicU64, +} + +impl Flash { + /// Format a fresh two-position store from a decoded payload. + pub fn new(payload: &[u8]) -> io::Result { + if payload.len() > PART_SIZE { + return Err(io::Error::new( + io::ErrorKind::InvalidData, + "Fuel PROM payload exceeds 8 MiB", + )); + } + let mut data = vec![0xff; FLASH_SIZE]; + data[..payload.len()].copy_from_slice(payload); + let label = PART_SIZE + 0x1e0000; + data[label + 0x10..label + 0x14].copy_from_slice(&0x504c4f49u32.to_be_bytes()); + data[label + 0x14..label + 0x18].copy_from_slice(&1u32.to_be_bytes()); + data[label + 0x18..label + 0x1c].copy_from_slice(&1u32.to_be_bytes()); + configure(&mut data); + Ok(Self { + st: Mutex::new(State::new(data)), + gen: AtomicU64::new(0), + }) + } + + /// Load an exact 16 MiB store, or seed a missing/empty one. Wrong sizes are + /// refused without writes. Reopening reapplies Fuel's configuration policy. + /// Completed guest programming and erasure write through to this file. + pub fn with_file(payload: &[u8], path: impl AsRef) -> io::Result { + let mut data = Self::new(payload)?.st.into_inner().data; + let mut backing = FlashBacking::open(path, &mut data).map_err(|error| { + if error.kind() == io::ErrorKind::InvalidData { + io::Error::new( + io::ErrorKind::InvalidData, + "Fuel flash store must be exactly 16 MiB (or empty)", + ) + } else { + error + } + })?; + configure(&mut data); + // Opening an existing store loads it before Fuel's configuration + // fields are refreshed; an empty store was seeded from this same data. + backing.write_range(0, &data)?; + let flash = Self { + st: Mutex::new(State { + data, + parts: std::array::from_fn(|_| AmdCore::default()), + backing: Some(backing), + }), + gen: AtomicU64::new(0), + }; + Ok(flash) + } + + pub fn contents(&self) -> Vec { + self.st.lock().data.clone() + } + pub fn generation(&self) -> u64 { + self.gen.load(Ordering::Acquire) + } + pub fn reset(&self) { + let mut st = self.st.lock(); + for part in &mut st.parts { + part.reset(); + } + self.gen.fetch_add(1, Ordering::Release); + } + /// Advance the model clock independently of flash reads. + pub fn advance_ns(&self, ns: u64) { + let mut st = self.st.lock(); + for part in 0..2 { + let selecting = st.parts[part].selecting(); + let complete = st.parts[part].advance_ns(ns); + if selecting && !st.parts[part].selecting() { + self.gen.fetch_add(1, Ordering::Release); + } + if !complete { + continue; + } + let sectors: Vec<_> = st.parts[part].selected_sectors().collect(); + let success = match st.erase(part, §ors) { + Ok(_) => true, + Err(e) => { + log::error!("bedrock-flash: position {part} erase backing write failed: {e}"); + false + } + }; + st.parts[part].finish_erase(success); + self.gen.fetch_add(1, Ordering::Release); + } + } + pub fn register_locks(self: &Arc) { + let me = self.clone(); + crate::locks::register_lock_fn("bedrock_flash::data", move || me.st.is_locked()); + } + /// Use the same shared flash for local and remote boot spaces. + pub fn boot_view(self: &Arc) -> FlashView { + FlashView { + flash: self.clone(), + range: None, + } + } + /// A contiguous executable alias. Offsets refer to the 16 MiB backing. + pub fn array_view(self: &Arc, offset: usize, len: usize) -> io::Result { + if offset.checked_add(len).is_none_or(|end| end > FLASH_SIZE) { + return Err(io::Error::new( + io::ErrorKind::InvalidInput, + "Fuel flash alias exceeds backing", + )); + } + Ok(FlashView { + flash: self.clone(), + range: Some((offset, len)), + }) + } +} +impl Resettable for Flash { + fn power_on(&self) { + self.reset(); + } +} + +pub struct FlashView { + flash: Arc, + range: Option<(usize, usize)>, +} +impl FlashView { + fn offset(&self, addr: u64, width: usize) -> Option { + if let Some((base, len)) = self.range { + let off = usize::try_from(addr).ok()?; + return off + .checked_add(width) + .filter(|end| *end <= len) + .map(|_| base + off); + } + let (base, within) = if addr < PART_SIZE as u64 { + (0, addr) + } else if addr >= SECOND_POSITION { + (PART_SIZE, addr - SECOND_POSITION) + } else { + return None; + }; + let within = usize::try_from(within).ok()?; + within + .checked_add(width) + .filter(|end| *end <= PART_SIZE) + .map(|_| base + within) + } + fn read(&self, addr: u64, width: usize) -> Option { + let off = self.offset(addr, width)?; + let mut st = self.flash.st.lock(); + if self.range.is_none() { + let part = off / PART_SIZE; + if !st.parts[part].array_reads() && (width != 2 || off & 1 != 0) { + return None; + } + if width == 2 && off & 1 == 0 { + if let Some(value) = st.parts[part].read16(off % PART_SIZE) { + return Some(value as u64); + } + } + } + Some( + st.data[off..off + width] + .iter() + .fold(0, |v, b| (v << 8) | *b as u64), + ) + } + fn refuse(&self, addr: u64, width: usize, value: u64) -> u32 { + crate::dlog!( + crate::devlog::LogModule::Bedrock, + "bedrock-flash: unsupported {width}-byte store {value:#x} at {addr:#x}; invalid bus cycle or executable view" + ); + BUS_ERR + } +} +impl BusDevice for FlashView { + fn read8(&self, a: u64) -> BusRead8 { + self.read(a, 1) + .map_or_else(BusRead8::err, |v| BusRead8::ok(v as u8)) + } + fn read16(&self, a: u64) -> BusRead16 { + self.read(a, 2) + .map_or_else(BusRead16::err, |v| BusRead16::ok(v as u16)) + } + fn read32(&self, a: u64) -> BusRead32 { + self.read(a, 4) + .map_or_else(BusRead32::err, |v| BusRead32::ok(v as u32)) + } + fn read64(&self, a: u64) -> BusRead64 { + self.read(a, 8).map_or_else(BusRead64::err, BusRead64::ok) + } + fn write8(&self, a: u64, v: u8) -> u32 { + self.refuse(a, 1, v as u64) + } + fn write16(&self, a: u64, v: u16) -> u32 { + self.refuse(a, 2, v as u64) + } + fn write32(&self, a: u64, v: u32) -> u32 { + self.refuse(a, 4, v as u64) + } + fn write64(&self, a: u64, v: u64) -> u32 { + if self.range.is_some() || a & 7 != 0 || v > u16::MAX as u64 { + return self.refuse(a, 8, v); + } + let (part, within) = if a < SECOND_POSITION { + (0, a) + } else { + (1, a - SECOND_POSITION) + }; + let Some(end) = within.checked_add(8) else { + return self.refuse(a, 8, v); + }; + if end > SECOND_POSITION { + return self.refuse(a, 8, v); + } + let offset = (within / 4) as usize; + let mut st = self.flash.st.lock(); + let effect = match st.parts[part].write16(offset, v as u16) { + Ok(effect) => effect, + Err(()) => { + self.flash.gen.fetch_add(1, Ordering::Release); + return BUS_ERR; + } + }; + if let Effect::Program { offset, value } = effect { + let backing = part * PART_SIZE + offset; + let old = u16::from_be_bytes([st.data[backing], st.data[backing + 1]]); + match st.replace(backing, &(old & value).to_be_bytes()) { + Ok(_) => {} + Err(e) => { + log::error!("bedrock-flash: program backing write at {backing:#x} failed: {e}"); + self.flash.gen.fetch_add(1, Ordering::Release); + return BUS_ERR; + } + } + } + self.flash.gen.fetch_add(1, Ordering::Release); + BUS_OK + } + #[cfg(feature = "jitv2")] + fn gen_ptr(&self, _addr: u64) -> *const AtomicU64 { + &self.flash.gen + } +} + +#[cfg(test)] +mod tests { + // Cases follow tests/qtest/sn1-bedrock-flash-test.c in + // jamesbraid/qemu-sgi-origami: program, sector-erase, erase-polling, + // reset-states and backing-file. + use super::*; + fn unlock(v: &FlashView, part: u64) { + assert_eq!(v.write64(part + 8 * 0x555, 0xaa), crate::traits::BUS_OK); + assert_eq!(v.write64(part + 8 * 0x2aa, 0x55), crate::traits::BUS_OK); + } + fn program(v: &FlashView, part: u64, off: u64, value: u16) { + unlock(v, part); + assert_eq!(v.write64(part + 8 * 0x555, 0xa0), crate::traits::BUS_OK); + assert_eq!( + v.write64(part + 4 * off, value as u64), + crate::traits::BUS_OK + ); + } + fn erase_select(v: &FlashView, part: u64, offset: u64) { + unlock(v, part); + assert_eq!(v.write64(part + 8 * 0x555, 0x80), BUS_OK); + unlock(v, part); + assert_eq!(v.write64(part + 4 * offset, 0x30), BUS_OK); + } + + /// The PROM reads CPU configuration and the volume label from its boot flash. + #[test] + fn the_seed_exposes_fuel_configuration_and_volume_label() { + let seed = vec![0x21; 0xb0]; + let f = Flash::new(&seed).unwrap(); + let bytes = f.contents(); + assert_eq!(sum(&bytes[0x60..0xb0]), sum(&seed[0x60..0xb0])); + assert_eq!(&bytes[0x64..0x68], &0x549aa985u32.to_be_bytes()); + assert_eq!(&bytes[0x74..0x78], &500_000_000u32.to_be_bytes()); + assert_eq!(&bytes[0x90..0x94], &1u32.to_be_bytes(), "CPU type: R14000"); + let label = PART_SIZE + 0x1e0000; + assert_eq!(&bytes[label..label + 16], &[0xff; 16]); + assert_eq!( + &bytes[label + 16..label + 28], + &[0x50, 0x4c, 0x4f, 0x49, 0, 0, 0, 1, 0, 0, 0, 1] + ); + assert!(bytes[label + 28..].iter().all(|b| *b == 0xff)); + assert!(Flash::new(&vec![0; PART_SIZE + 1]).is_err()); + } + /// The PROM programs through one view while executable aliases must + /// keep reading array bytes. + #[test] + fn programming_changes_shared_bytes_without_exposing_status_to_instruction_fetches() { + { + let f = Arc::new(Flash::new(&[]).unwrap()); + let local = f.boot_view(); + let remote = f.boot_view(); + let execute = f.array_view(0, PART_SIZE).unwrap(); + assert_eq!(local.write64(SECOND_POSITION, 0xf0), crate::traits::BUS_OK); + for part in [0, SECOND_POSITION] { + assert_eq!(local.write64(part + 8 * 0x55, 0x98), crate::traits::BUS_OK); + assert_eq!(remote.read16(part + 0x20).data, b'Q' as u16); + assert_eq!(remote.read16((SECOND_POSITION - part) + 0x20).data, 0xffff); + assert_eq!(execute.read16(0x20).data, 0xffff); + assert_eq!(remote.write64(part, 0xf0), crate::traits::BUS_OK); + assert_eq!(local.write64(part + 8 * 0x555, 0xaa), crate::traits::BUS_OK); + assert_eq!( + remote.write64(part + 8 * 0x2aa, 0x55), + crate::traits::BUS_OK + ); + assert_eq!(local.write64(part + 8 * 0x555, 0xa0), crate::traits::BUS_OK); + assert_eq!( + remote.write64(part + 4 * 0x200, 0x1234), + crate::traits::BUS_OK + ); + assert_eq!(local.read16(part + 0x200).data, 0x1234); + } + assert_eq!(execute.read16(0x200).data, 0x1234); + assert!(!local.read8(PART_SIZE as u64).is_ok()); + } + { + let f = Arc::new(Flash::new(&[]).unwrap()); + let v = f.boot_view(); + program(&v, 0, 0x200, 0x1234); + program(&v, 0, 0x200, 0xf0ff); + assert_eq!(v.read16(0x200).data, 0x1034); + unlock(&v, 0); + assert_eq!(v.write64(8 * 0x555, 0xa0), crate::traits::BUS_OK); + let gen = f.generation(); + for (addr, value) in [ + (0x800, 0x10000), + (0x804, 0), + (64 << 20, 0), + ] { + assert_eq!(v.write64(addr, value), BUS_ERR); + } + assert_eq!(v.write16(0x800, 0), BUS_ERR); + assert_eq!(f.generation(), gen); + assert_eq!(v.write64(0x800, 0x00ff), crate::traits::BUS_OK); + assert_eq!(v.read16(0x200).data, 0x0034); + let execute = f.array_view(0, PART_SIZE).unwrap(); + unlock(&v, 0); + assert_eq!(execute.write64(8 * 0x555, 0xa0), BUS_ERR); + assert_eq!(v.write64(8 * 0x555, 0xa0), crate::traits::BUS_OK); + assert_eq!(v.write64(0x808, 0x5555), crate::traits::BUS_OK); + assert_eq!(execute.read16(0x202).data, 0x5555); + } + { + let f = Arc::new(Flash::new(&[]).unwrap()); + let v = f.boot_view(); + assert_eq!(v.write64(0x550, 0x98), BUS_ERR); + assert_eq!(v.read16(0x20).data, 0xffff); + assert_eq!(v.write64(8 * 0x55, 0x98), BUS_OK); + let expected = [ + 0x51, 0x52, 0x59, 2, 0, 0, 0, 0, 0, 0, 0, 0x27, 0x36, 0, 0, 3, 0, 9, 0, 4, 0, 4, 0, 23, + 1, 0, 0, 0, 1, 0x7f, 0, 0, 1, + ]; + for (index, value) in expected.into_iter().enumerate() { + assert_eq!(v.read16(((index + 0x10) * 2) as u64).data, value); + } + assert_eq!(v.read16(0x100).data, 0); + assert!(!v.read8(0x20).is_ok()); + assert!(!v.read16(0x21).is_ok()); + f.reset(); + assert_eq!(v.read16(0x20).data, 0xffff); + } + } + /// The PROM polls erase completion and can reset the chip before + /// selected sectors change. + #[test] + fn erase_completes_only_for_selected_sectors_and_reset_cancels_it() { + let f = Arc::new(Flash::new(&[]).unwrap()); + let v = f.boot_view(); + program(&v, 0, 0x200, 0); + program(&v, 0, 0x10000, 0); + erase_select(&v, 0, 0x200); + assert!(!v.read8(0x200).is_ok()); + let a = v.read16(0x200).data; + let b = v.read16(0x200).data; + assert_eq!((a ^ b) & 0x44, 0x44); + let a = v.read16(0x10000).data; + let b = v.read16(0x10000).data; + assert_eq!((a ^ b) & 0x44, 0x40); + f.advance_ns(40_000); + assert_eq!(v.write64(4 * 0x10000, 0x30), BUS_OK); + f.advance_ns(40_000); + assert_eq!(v.read16(0x10000).data & 8, 0); + assert_eq!(v.write64(SECOND_POSITION + 8 * 0x55, 0x98), BUS_OK); + assert_eq!(v.read16(SECOND_POSITION + 0x20).data, b'Q' as u16); + f.advance_ns(10_000); + assert_eq!(v.read16(0x200).data & 8, 8); + assert_eq!(v.write64(0, 0xf0), BUS_OK); + assert_eq!(v.read16(0x200).data & 0x80, 0); + f.reset(); + f.advance_ns(u64::MAX); + assert_eq!(v.read16(0x200).data, 0); + assert_eq!(v.read16(0x10000).data, 0); + assert_eq!(v.read16(SECOND_POSITION + 0x20).data, 0xffff); + erase_select(&v, 0, 0x200); + assert_eq!(v.write64(0, 0xf0), BUS_OK); + f.advance_ns(u64::MAX); + assert_eq!(v.read16(0x200).data, 0); + erase_select(&v, 0, 0x200); + assert_eq!(v.write64(4 * 0x10000, 0x30), BUS_OK); + f.advance_ns(50_000 + 512_000_000); + assert_eq!(v.read16(0x200).data, 0xffff); + assert_eq!(v.read16(0x10000).data, 0xffff); + unlock(&v, 0); + assert_eq!(v.write64(8 * 0x555, 0xa0), BUS_OK); + f.reset(); + assert_eq!(v.write64(4 * 0x200, 0x1234), BUS_ERR); + assert_eq!(v.read16(0x200).data, 0xffff); + program(&v, 0, 0x200, 0); + program(&v, 0, 0x10000, 0x3456); + erase_select(&v, 0, 0x200); + f.advance_ns(50_000 + 512_000_000); + assert_eq!(v.read16(0x200).data, 0xffff); + assert_eq!(v.read16(0x10000).data, 0x3456); + } + /// Firmware updates must persist across emulator restarts without + /// overwriting a supplied image. + #[test] + fn completed_programming_and_erases_survive_reopening_the_backing_file() { + { + let path = std::env::temp_dir().join(format!("iris-bedrock-store-{}", std::process::id())); + let mut bytes = vec![0x5a; FLASH_SIZE]; + bytes[PART_SIZE] = 0xc7; + std::fs::write(&path, &bytes).unwrap(); + let f = Flash::with_file(&[0x11], &path).unwrap(); + assert_eq!(f.contents()[0], 0x5a); + assert_eq!(f.contents()[PART_SIZE], 0xc7); + assert_eq!(std::fs::read(&path).unwrap(), f.contents()); + drop(f); + std::fs::write(&path, [1, 2, 3]).unwrap(); + assert!(Flash::with_file(&[], &path).is_err()); + assert_eq!(std::fs::read(&path).unwrap(), [1, 2, 3]); + std::fs::write(&path, []).unwrap(); + let g = Flash::with_file(&[0x34], &path).unwrap(); + assert_eq!(g.contents()[0], 0x34); + assert_eq!(std::fs::metadata(&path).unwrap().len(), FLASH_SIZE as u64); + drop(g); + std::fs::remove_file(path).unwrap(); + } + { + let path = std::env::temp_dir().join(format!("iris-amd-persist-{}", std::process::id())); + std::fs::write(&path, []).unwrap(); + let f = Arc::new(Flash::with_file(&[], &path).unwrap()); + let v = f.boot_view(); + program(&v, 0, 0x200, 0x1234); + program(&v, SECOND_POSITION, (PART_SIZE - 2) as u64, 0x5678); + drop(v); + drop(f); + let f = Arc::new(Flash::with_file(&[], &path).unwrap()); + let v = f.boot_view(); + assert_eq!(v.read16(0x200).data, 0x1234); + assert_eq!( + v.read16(SECOND_POSITION + PART_SIZE as u64 - 2).data, + 0x5678 + ); + unlock(&v, SECOND_POSITION); + assert_eq!(v.write64(SECOND_POSITION + 8 * 0x555, 0x80), BUS_OK); + unlock(&v, SECOND_POSITION); + assert_eq!( + v.write64(SECOND_POSITION + 4 * (PART_SIZE as u64 - 2), 0x30), + BUS_OK + ); + f.advance_ns(50_000 + 512_000_000); + drop(v); + drop(f); + let f = Arc::new(Flash::with_file(&[], &path).unwrap()); + let v = f.boot_view(); + assert_eq!(v.read16(0x200).data, 0x1234); + assert_eq!( + v.read16(SECOND_POSITION + PART_SIZE as u64 - 2).data, + 0xffff + ); + std::fs::remove_file(path).unwrap(); + } + } + + /// A host write failure must preserve flash contents and remain visible + /// to the guest. + #[test] + fn failed_backing_writes_leave_array_bytes_unchanged() { + { + let f = Arc::new(Flash::new(&[]).unwrap()); + // A read-only descriptor makes writes fail without a special backend. + let path = std::env::temp_dir().join(format!("iris-amd-read-only-{}", std::process::id())); + std::fs::write(&path, []).unwrap(); + f.st.lock().backing = Some(FlashBacking::from_file( + std::fs::File::open(&path).unwrap(), + FLASH_SIZE, + )); + let v = f.boot_view(); + unlock(&v, 0); + assert_eq!(v.write64(8 * 0x555, 0xa0), BUS_OK); + assert_eq!(v.write64(4 * 0x200, 0x1234), BUS_ERR); + assert_eq!(v.read16(0x200).data, 0xffff); + std::fs::remove_file(path).unwrap(); + } + { + let f = Arc::new(Flash::new(&[]).unwrap()); + let v = f.boot_view(); + program(&v, 0, 0x200, 0); + let path = + std::env::temp_dir().join(format!("iris-amd-erase-failure-{}", std::process::id())); + std::fs::write(&path, []).unwrap(); + f.st.lock().backing = Some(FlashBacking::from_file( + std::fs::File::open(&path).unwrap(), + FLASH_SIZE, + )); + erase_select(&v, 0, 0x200); + f.advance_ns(50_000 + 512_000_000); + assert_eq!(v.read16(0x200).data & 0x20, 0x20); + assert_eq!(f.contents()[0x200], 0); + assert_eq!(v.write64(0, 0xf0), BUS_OK); + assert_eq!(v.read16(0x200).data, 0); + std::fs::remove_file(path).unwrap(); + } + } + +} diff --git a/src/dev/bedrock_io.rs b/src/dev/bedrock_io.rs new file mode 100644 index 000000000..4b3e659c3 --- /dev/null +++ b/src/dev/bedrock_io.rs @@ -0,0 +1,811 @@ +//! Bedrock IO routes programmed PIO windows to PCI widgets, applies inbound +//! permission checks to DMA requests and exposes two block-transfer engines. +//! Guests configure the widget masks before touching devices; the BTE moves +//! 128-byte lines and writes completion state through the configured target. +//! +//! # What is approximated here +//! +//! BTE transfers and their completion callbacks run synchronously when a +//! start is written. Timing and unpublished register behavior are not modeled. +//! +//! The plain register words sit in a sparse map keyed by offset, since the +//! 0x30010-byte block holds only a few dozen of them. +//! +//! Ported from hw/misc/sgi-bedrock-io.c in jamesbraid/qemu-sgi-origami. + +/// Length of the block-relative register aperture. +pub const SIZE: u64 = 0x30010; + +/// The public Linux Hub I/O protection-override key literal. +/// +/// Linux `arch/mips/include/asm/sn/sn0/hubio.h` at +/// `db2ddb87143519e20a95aa36c60b36107b736a58` defines +/// `IIO_PROTECT_OVRRD_KEY` as this value (lines 31, 47). This establishes +/// the literal's provenance only; it does not establish a register mapping. +pub const IIO_PROTECT_OVRRD_KEY: u64 = 0x53474972756c6573; + +pub mod reg { + pub const DEVICE_ID: u64 = 0x0; + pub const PROTECTION_KEY: u64 = 0x108; + pub const PIO_PERMISSION: u64 = 0x100; + pub const INBOUND_PERMISSION: u64 = 0x110; + pub const INBOUND_DEVICE_PERMISSION: u64 = 0x120; + pub const XIO_STATUS: u64 = 0x128; + pub const LINK_LOG: u64 = 0x130; + pub const LINK_STATUS: u64 = 0x138; + pub const BTE1_BASE: u64 = 0x10000; + pub const BTE_REGISTER_SIZE: u64 = 0x30; + pub const READ_ONLY: u64 = 0x8; + pub const BTE2_BASE: u64 = 0x20000; + pub const WIDGET_STATUS: u64 = 0x20; + pub const WIDGET_PERMISSION: u64 = 0x118; + pub const REGISTER_0150: u64 = 0x150; + pub const REGISTER_0158: u64 = 0x158; + pub const ITTE_BASE: u64 = 0x160; + pub const ITTE_LAST: u64 = 0x190; + pub const REGISTER_0198_BASE: u64 = 0x198; + pub const REGISTER_01D8_LAST: u64 = 0x1d8; + pub const READ_ONLY_ALIAS: u64 = 0x1f8; + pub const REGISTER_01E0: u64 = 0x1e0; + pub const REGISTER_01E8: u64 = 0x1e8; + pub const REGISTER_01F0: u64 = 0x1f0; + pub const REGISTER_0200: u64 = 0x200; + pub const REGISTER_03A8: u64 = 0x3a8; + pub const REGISTER_03B0: u64 = 0x3b0; + pub const REGISTER_03B8: u64 = 0x3b8; + pub const REGISTER_03C0: u64 = 0x3c0; + pub const RETAINED_WINDOW_BASE: u64 = 0x308; + pub const RETAINED_WINDOW_LAST: u64 = 0x348; + pub const RETAINED_CONFIG_BASE: u64 = 0x400; + pub const RETAINED_CONFIG_LAST: u64 = 0x5d8; + pub const GFX_MAP_BASE: u64 = 0x140; +} +pub mod bte_reg { + pub const STATUS: u64 = 0x0; + pub const SOURCE: u64 = 0x8; + pub const DESTINATION: u64 = 0x10; + pub const CONTROL: u64 = 0x18; + pub const NOTIFICATION: u64 = 0x20; + pub const INTERRUPT: u64 = 0x28; +} + +use crate::traits::{BusDevice, BusRead64, Resettable, BUS_ERR, BUS_OK}; +use parking_lot::Mutex; +use std::collections::{BTreeMap, HashSet}; +use std::sync::Arc; +use std::thread::ThreadId; +const ID: u64 = (2 << 28) | (0xc110 << 12) | (0x36 << 1); +const ADDR_MASK: u64 = 0xffffffff80; +const BUSY: u64 = 1 << 20; +const ERROR: u64 = 1 << 16; +#[derive(Clone, Copy, Default)] +struct Bte { + status: u64, + source: u64, + destination: u64, + control: u64, + notification: u64, + interrupt: u64, +} +struct Regs { + words: BTreeMap, + bte: [Bte; 2], + node: u8, + peer: bool, +} +impl Default for Regs { + fn default() -> Self { + let mut words = BTreeMap::new(); + for (a, v) in [ + (0x20, 0xa), + (0x100, !0), + (0x110, !0), + (0x118, !0), + (0x120, !0), + (0x128, 0x3fc0fff6403 & !(1 << 13)), + (0x1e8, 0x111), + ] { + words.insert(a, v); + } + Self { + words, + bte: [Bte::default(); 2], + node: 0, + peer: false, + } + } +} +// Only a synchronous callback on the same thread is recursive. DMA workers +// on other threads may call the thread-safe downstream bus independently. +struct InboundAccess<'a> { + active: &'a Mutex>, + thread: ThreadId, +} +impl Drop for InboundAccess<'_> { + fn drop(&mut self) { + self.active.lock().remove(&self.thread); + } +} + +pub struct BedrockIo { + inbound_threads: Mutex>, + regs: Arc>, + widgets: Mutex<[Option>; 16]>, + memory: Mutex>>, + hub: Mutex>>, +} +impl Default for BedrockIo { + fn default() -> Self { + Self::new() + } +} +impl BedrockIo { + pub fn new() -> Self { + let regs = Arc::new(Mutex::new(Regs::default())); + crate::locks::register_mutex("bedrock-io", ®s); + Self { + regs, + inbound_threads: Mutex::new(HashSet::new()), + widgets: Mutex::new(std::array::from_fn(|_| None)), + memory: Mutex::new(None), + hub: Mutex::new(None), + } + } + pub fn connect_widget(&self, widget: u8, dev: Arc) { + assert!(widget < 16); + self.widgets.lock()[widget as usize] = Some(dev); + self.set_xio_peer(true); + } + pub fn set_xio_peer(&self, peer: bool) { + let mut r = self.regs.lock(); + r.peer = peer; + if peer { + *r.words.entry(reg::XIO_STATUS).or_default() |= 1 << 13; + } else { + *r.words.entry(reg::XIO_STATUS).or_default() &= !(1 << 13); + } + } + pub fn connect_memory(&self, dev: Arc) { + *self.memory.lock() = Some(dev); + } + pub fn connect_hub(&self, dev: Arc) { + *self.hub.lock() = Some(dev); + } + pub fn set_node_number(&self, node: u8) { + self.regs.lock().node = node; + } + pub fn gfx_map(&self, entry: usize) -> u64 { + assert!(entry < 2); + self.word(reg::GFX_MAP_BASE + entry as u64 * 8) + } + fn word(&self, a: u64) -> u64 { + self.regs.lock().words.get(&a).copied().unwrap_or(0) + } + pub fn disable_device(&self, widget: u8, device: u8) { + if (8..16).contains(&widget) && device < 8 { + *self.regs.lock().words.entry(reg::INBOUND_DEVICE_PERMISSION).or_default() &= + !(1 << ((widget - 8) * 8 + device)); + } + } + fn permitted(r: &Regs, widget: u8, device: u8) -> bool { + widget < 16 + && device < 8 + && r.words.get(®::WIDGET_PERMISSION).copied().unwrap_or(0) & (1 << widget) != 0 + && (widget < 8 + || r.words.get(®::INBOUND_DEVICE_PERMISSION).copied().unwrap_or(0) & (1 << ((widget - 8) * 8 + device)) + != 0) + } + fn retained(a: u64) -> bool { + matches!( + a, + 0x208 + | 0x210 + | 0x218 + | 0x220 + | 0x228 + | 0x300 + | reg::GFX_MAP_BASE + | 0x148 + | 0x388 + | 0x390 + | 0x398 + | 0x3a0 + | 0x600 + | 0x608 + | 0x610 + | 0x30000 + | 0x30008 + ) || (reg::RETAINED_WINDOW_BASE..reg::RETAINED_WINDOW_LAST).contains(&a) + || (reg::RETAINED_CONFIG_BASE..=reg::RETAINED_CONFIG_LAST).contains(&a) + } + fn mask(a: u64) -> Option { + Some(match a { + reg::WIDGET_STATUS => 0x7fe01f, + reg::PIO_PERMISSION | reg::INBOUND_PERMISSION | reg::WIDGET_PERMISSION | reg::INBOUND_DEVICE_PERMISSION | reg::REGISTER_0150 | reg::REGISTER_0158 => !0, + reg::XIO_STATUS => 0x3ff0fff7d3f, + reg::LINK_STATUS => 0x101ff7f, + reg::ITTE_BASE..=reg::ITTE_LAST => 0x1f1f, + reg::REGISTER_0198_BASE..=reg::REGISTER_01D8_LAST => !0, + reg::REGISTER_01E0 => 0x3ffffff, + reg::REGISTER_01E8 => 0x1111, + reg::REGISTER_01F0 => 0x0fffffff03ffffff, + reg::REGISTER_0200 => 0x10f, + reg::REGISTER_03A8 => 0x13ffffffff30f3, + reg::REGISTER_03B0 => 0x1ff0f, + reg::REGISTER_03B8 => 0xff, + reg::REGISTER_03C0 => 0xffffff, + _ if Self::retained(a) => !0, + _ => return None, + }) + } + fn bte_at(a: u64) -> Option<(usize, u64)> { + if (reg::BTE1_BASE..reg::BTE1_BASE + reg::BTE_REGISTER_SIZE).contains(&a) { + Some((0, a - reg::BTE1_BASE)) + } else if (reg::BTE2_BASE..reg::BTE2_BASE + reg::BTE_REGISTER_SIZE).contains(&a) { + Some((1, a - reg::BTE2_BASE)) + } else { + None + } + } + fn target(&self, offset: u64) -> Option<(u8, u64)> { + if offset < 0x10000000 { + Some(((offset >> 24) as u8, offset & 0xffffff)) + } else if offset >= 0x20000000 { + let entry = (offset >> 29) - 1; + if entry >= 7 { + return None; + } + let itte = self.word(reg::ITTE_BASE + entry * 8); + Some(( + ((itte >> 8) & 15) as u8, + ((itte & 31) << 29) + | (offset & 0x1fffffff) + | if itte & (1 << 12) != 0 { 1 << 47 } else { 0 }, + )) + } else { + None + } + } + pub fn pio_read(&self, offset: u64, size: u8) -> BusRead64 { + self.pio(offset, size, None) + } + pub fn pio_write(&self, offset: u64, size: u8, value: u64) -> u32 { + self.pio(offset, size, Some(value)).status + } + fn pio(&self, offset: u64, size: u8, value: Option) -> BusRead64 { + let Some((widget, target)) = self.target(offset) else { + return BusRead64::err(); + }; + { + let r = self.regs.lock(); + if r.words.get(&0x100).copied().unwrap_or(0) & 1 == 0 + || r.words.get(&0x110).copied().unwrap_or(0) & (1 << widget) == 0 + { + return BusRead64::err(); + } + } + if !matches!(size, 1 | 2 | 4 | 8) { + return BusRead64::err(); + } + if widget == 1 && target == 0 { + return BusRead64::ok(ID); + } + let dev = self.widgets.lock()[widget as usize].clone(); + Self::access(dev, target, size, value) + } + /// Device None means every possible DMA master must be permitted. + pub fn inbound_read( + &self, + widget: u8, + device: Option, + address: u64, + size: u8, + ) -> BusRead64 { + self.inbound(widget, device, address, size, None) + } + pub fn inbound_write( + &self, + widget: u8, + device: Option, + address: u64, + size: u8, + value: u64, + ) -> u32 { + self.inbound(widget, device, address, size, Some(value)) + .status + } + fn inbound( + &self, + widget: u8, + device: Option, + address: u64, + size: u8, + value: Option, + ) -> BusRead64 { + if address >= (1 << 48) || size as u64 > (1 << 48) - address { + return BusRead64::err(); + } + { + let r = self.regs.lock(); + let allowed = if let Some(device) = device { + Self::permitted(&r, widget, device) + } else { + (0..8).all(|d| Self::permitted(&r, widget, d)) + }; + if !allowed { + crate::dlog!(crate::devlog::LogModule::Bedrock, + "Bedrock IO inbound access refused from widget {widget} device {device:?}" + ); + return BusRead64::err(); + } + } + let thread = std::thread::current().id(); + if !self.inbound_threads.lock().insert(thread) { + crate::dlog!(crate::devlog::LogModule::Bedrock, "Bedrock IO reentrant inbound packet refused"); + return BusRead64::err(); + } + let _access = InboundAccess { + active: &self.inbound_threads, + thread, + }; + if address & (1 << 47) != 0 { + let node = ((address >> 39) & 255) as u8; + let window = address & ((1 << 39) - 1); + if node != self.regs.lock().node || window & !0x7fffff != 0x1800000 { + return BusRead64::err(); + } + let dev = self.hub.lock().clone(); + Self::access(dev, window & 0x7fffff, size, value) + } else { + let dev = self.memory.lock().clone(); + Self::access(dev, address, size, value) + } + } + fn access(dev: Option>, a: u64, size: u8, value: Option) -> BusRead64 { + let Some(dev) = dev else { + return BusRead64::err(); + }; + if let Some(v) = value { + let status = match size { + 1 => dev.write8(a, v as u8), + 2 => dev.write16(a, v as u16), + 4 => dev.write32(a, v as u32), + 8 => dev.write64(a, v), + _ => BUS_ERR, + }; + BusRead64 { status, data: 0 } + } else { + match size { + 1 => { + let r = dev.read8(a); + BusRead64 { + status: r.status, + data: r.data as u64, + } + } + 2 => { + let r = dev.read16(a); + BusRead64 { + status: r.status, + data: r.data as u64, + } + } + 4 => { + let r = dev.read32(a); + BusRead64 { + status: r.status, + data: r.data as u64, + } + } + 8 => dev.read64(a), + _ => BusRead64::err(), + } + } + } + fn run_bte(&self, n: usize, mut b: Bte) { + let memory = self.memory.lock().clone(); + let count = b.status & 0xffff; + let bytes = count * 128; + let overflow = (b.control & 1 == 0 && b.source > (1 << 40) - bytes) + || b.destination > (1 << 40) - bytes; + let mut remaining = count; + if overflow { + b.status = remaining | ERROR; + } else { + while remaining != 0 { + let mut line = [0u64; 16]; + let mut status = BUS_OK; + if b.control & 1 == 0 { + for (i, v) in line.iter_mut().enumerate() { + let r = Self::access(memory.clone(), b.source + i as u64 * 8, 8, None); + status = r.status; + if status != BUS_OK { + break; + } + *v = r.data; + } + } + if status == BUS_OK { + for (i, v) in line.into_iter().enumerate() { + status = + Self::access(memory.clone(), b.destination + i as u64 * 8, 8, Some(v)) + .status; + if status != BUS_OK { + break; + } + } + } + if status != BUS_OK { + break; + } + remaining -= 1; + b.source = (b.source + 128) & ADDR_MASK; + b.destination = (b.destination + 128) & ADDR_MASK; + } + b.status = if remaining == 0 { 0 } else { remaining | ERROR }; + } + self.regs.lock().bte[n] = b; + if b.control & 0x10 != 0 { + if Self::access(memory, b.notification, 8, Some(b.status)).status != BUS_OK { + crate::dlog!(crate::devlog::LogModule::Bedrock, "Bedrock IO BTE notification write failed"); + } + } else { + let node = ((b.interrupt >> 1) & 255) as u8; + if node != self.regs.lock().node { + crate::dlog!(crate::devlog::LogModule::Bedrock, "Bedrock IO BTE interrupt reaches no node"); + return; + } + let hub = self.hub.lock().clone(); + let offset = if b.interrupt & 1 != 0 { 0x200090 } else { 0x90 }; + if Self::access(hub, offset, 8, Some(0x100 | ((b.interrupt >> 16) & 127))).status + != BUS_OK + { + crate::dlog!(crate::devlog::LogModule::Bedrock, "Bedrock IO BTE completion interrupt failed"); + } + } + } +} +impl Resettable for BedrockIo { + fn power_on(&self) { + let mut r = self.regs.lock(); + let peer = r.peer; + let node = r.node; + *r = Regs::default(); + r.node = node; + r.peer = peer; + if peer { + *r.words.entry(reg::XIO_STATUS).or_default() |= 1 << 13; + } + } +} +impl BusDevice for BedrockIo { + fn read64(&self, a: u64) -> BusRead64 { + if a & 7 != 0 { + return BusRead64::err(); + } + let mut r = self.regs.lock(); + if let Some((n, a)) = Self::bte_at(a) { + let b = &mut r.bte[n]; + return BusRead64::ok(match a { + bte_reg::STATUS => b.status, + bte_reg::SOURCE => b.source, + bte_reg::DESTINATION => b.destination, + bte_reg::CONTROL => { + b.status &= !BUSY; + b.control + } + bte_reg::NOTIFICATION => b.notification, + bte_reg::INTERRUPT => b.interrupt, + _ => 0, + }); + } + let v = match a { + reg::DEVICE_ID => ID, + reg::READ_ONLY | 0x1f8 => 0, + reg::LINK_STATUS => r.words.get(&a).copied().unwrap_or(0) & !(1 << 28), + reg::LINK_LOG => r.words.get(&a).copied().unwrap_or(0), + _ if Self::mask(a).is_some() => r.words.get(&a).copied().unwrap_or(0), + _ => { + crate::dlog!(crate::devlog::LogModule::Bedrock, "Bedrock IO unsupported read {a:#x}"); + 0 + } + }; + BusRead64::ok(v) + } + fn write64(&self, a: u64, v: u64) -> u32 { + if a & 7 != 0 { + return BUS_ERR; + } + let mut r = self.regs.lock(); + if let Some((n, offset)) = Self::bte_at(a) { + let b = &mut r.bte[n]; + let mut run = false; + match offset { + bte_reg::STATUS => b.status = v & (0xffff | BUSY), + bte_reg::SOURCE => b.source = v & ADDR_MASK, + bte_reg::DESTINATION => b.destination = v & ADDR_MASK, + bte_reg::CONTROL => { + b.control = v & 0x111; + if b.control & 0x100 != 0 { + crate::dlog!(crate::devlog::LogModule::Bedrock, "Bedrock IO BTE poison unsupported"); + } else if b.status & BUSY != 0 { + if b.status & 0xffff == 0 { + b.status = 0; + crate::dlog!(crate::devlog::LogModule::Bedrock, "Bedrock IO zero-count BTE start"); + } else { + run = true; + } + } + } + bte_reg::NOTIFICATION => b.notification = v & ADDR_MASK, + bte_reg::INTERRUPT => b.interrupt = v & 0x7f01ff, + _ => {} + } + let b = *b; + drop(r); + if run { + self.run_bte(n, b); + } + return BUS_OK; + } + match a { + reg::PROTECTION_KEY => { + if v == IIO_PROTECT_OVRRD_KEY { + *r.words.entry(0x100).or_default() |= 1; + } + } + reg::INBOUND_DEVICE_PERMISSION | reg::REGISTER_01E8 => { + *r.words.entry(a).or_default() |= v & Self::mask(a).unwrap(); + } + reg::XIO_STATUS => { + let v = v & Self::mask(a).unwrap() & if r.peer { !0 } else { !(1 << 13) }; + r.words.insert(a, v); + } + reg::LINK_LOG => { + if v == 0 { + r.words.insert(a, 0); + } else { + crate::dlog!(crate::devlog::LogModule::Bedrock, "Bedrock IO nonzero link-log write ignored"); + } + } + reg::READ_ONLY_ALIAS | reg::DEVICE_ID | reg::READ_ONLY => { + crate::dlog!(crate::devlog::LogModule::Bedrock, "Bedrock IO read-only or unsupported clear write {a:#x}"); + } + _ => { + if let Some(mask) = Self::mask(a) { + r.words.insert(a, v & mask); + } else { + crate::dlog!(crate::devlog::LogModule::Bedrock, "Bedrock IO unsupported write {a:#x} = {v:#x}"); + } + } + } + BUS_OK + } +} + +#[cfg(test)] +mod tests { + // Cases follow tests/qtest/sn1-bedrock-io-test.c in + // jamesbraid/qemu-sgi-origami: inbound-protection, bte and + // bte-error-termination. + use super::*; + #[derive(Default)] + struct Memory { + words: Mutex>, + fail_read: Mutex>, + fail_write: Mutex>, + } + impl BusDevice for Memory { + fn read64(&self, a: u64) -> BusRead64 { + if *self.fail_read.lock() == Some(a) { + return BusRead64::err(); + } + BusRead64::ok(self.words.lock().get(&a).copied().unwrap_or(0)) + } + fn write64(&self, a: u64, v: u64) -> u32 { + if *self.fail_write.lock() == Some(a) { + return BUS_ERR; + } + self.words.lock().insert(a, v); + BUS_OK + } + } + fn launch(d: &BedrockIo, base: u64, count: u64, src: u64, dst: u64, notify: u64, control: u64) { + d.write64(base, count | BUSY); + d.write64(base + 8, src); + d.write64(base + 0x10, dst); + d.write64(base + 0x20, notify); + d.write64(base + 0x18, control); + } + struct Reentrant(std::sync::Weak); + impl BusDevice for Reentrant { + fn read64(&self, a: u64) -> BusRead64 { + self.0.upgrade().unwrap().inbound_read(15, Some(0), a, 8) + } + } + struct Busy; + impl BusDevice for Busy { + fn read64(&self, _: u64) -> BusRead64 { + BusRead64::busy() + } + fn write64(&self, _: u64, _: u64) -> u32 { + crate::traits::BUS_BUSY + } + } + struct BlockingMemory { + entered: std::sync::mpsc::Sender<()>, + release: Mutex>, + words: Mutex>, + } + impl BusDevice for BlockingMemory { + fn write64(&self, address: u64, value: u64) -> u32 { + if address == 0x1000 { + self.entered.send(()).unwrap(); + self.release + .lock() + .recv_timeout(std::time::Duration::from_secs(5)) + .unwrap(); + } + self.words.lock().insert(address, value); + BUS_OK + } + } + + /// The guest must enable widget and requester masks before PIO or inbound DMA succeeds. + #[test] + fn the_io_block_checks_pio_and_inbound_requester_permissions() { + { + let d = BedrockIo::new(); + let m = Arc::new(Memory::default()); + d.connect_widget(8, m.clone()); + d.write64(reg::ITTE_BASE, 0x802); + assert_eq!(d.pio_write(0x20000080, 8, 0xabcd), BUS_OK); + assert_eq!(m.read64(0x40000080).data, 0xabcd); + assert_eq!(d.pio_read(0x20000080, 8).data, 0xabcd); + d.write64(reg::INBOUND_PERMISSION, 0); + assert_eq!(d.pio_read(0x20000080, 8).status, BUS_ERR); + d.write64(reg::INBOUND_PERMISSION, !0); + d.write64(reg::PIO_PERMISSION, 0); + assert_eq!(d.pio_read(0x20000080, 8).status, BUS_ERR); + d.write64(reg::PROTECTION_KEY, IIO_PROTECT_OVRRD_KEY); + assert_eq!(d.pio_read(0x20000080, 8).status, BUS_OK); + assert_eq!(d.pio_read(0x10000000, 8).status, BUS_ERR); + d.power_on(); + assert_eq!(d.read64(reg::PIO_PERMISSION).data, !0); + assert_eq!(d.read64(reg::REGISTER_01E8).data, 0x111); + + } + { + let d = BedrockIo::new(); + let m = Arc::new(Memory::default()); + let h = Arc::new(Memory::default()); + d.connect_memory(m.clone()); + d.connect_hub(h.clone()); + assert_eq!(d.inbound_write(14, Some(2), 0x80, 8, 42), BUS_OK); + assert_eq!(m.read64(0x80).data, 42); + d.disable_device(14, 2); + assert_eq!(d.inbound_read(14, Some(2), 0x80, 8).status, BUS_ERR); + assert_eq!(d.inbound_read(14, None, 0x80, 8).status, BUS_ERR); + assert_eq!(d.inbound_read(14, Some(1), 0x80, 8).status, BUS_OK); + let a = (1 << 47) | 0x1800090; + assert_eq!(d.inbound_write(15, Some(0), a, 8, 0x123), BUS_OK); + assert_eq!(h.read64(0x90).data, 0x123); + assert_eq!( + d.inbound_read(15, Some(0), a | (1 << 39), 8).status, + BUS_ERR + ); + assert_eq!(d.inbound_read(15, Some(0), 1 << 48, 8).status, BUS_ERR); + assert_eq!(d.inbound_read(15, Some(0), a + 0x800000, 8).status, BUS_ERR); + } + } + /// The guest observes BTE line copies and zero-fill through its completion registers. + #[test] + fn the_bte_copies_and_zeroes_lines_before_completion() { + let d = BedrockIo::new(); + let m = Arc::new(Memory::default()); + let h = Arc::new(Memory::default()); + d.connect_memory(m.clone()); + d.connect_hub(h.clone()); + for i in 0..32 { + m.write64(0x1000 + i * 8, 0xfeed0000 | i); + } + launch(&d, reg::BTE1_BASE, 2, 0x1000, 0x2000, 0x3000, 0x10); + for i in 0..32 { + assert_eq!(m.read64(0x2000 + i * 8).data, 0xfeed0000 | i); + } + assert_eq!(d.read64(reg::BTE1_BASE + bte_reg::STATUS).data, 0); + assert_eq!(d.read64(reg::BTE1_BASE + bte_reg::SOURCE).data, 0x1100); + assert_eq!(m.read64(0x3000).data, 0); + d.write64(reg::BTE2_BASE + bte_reg::INTERRUPT, 53 << 16); + launch(&d, reg::BTE2_BASE, 1, 0, 0x2000, 0, 1); + for i in 0..16 { + assert_eq!(m.read64(0x2000 + i * 8).data, 0); + } + assert_eq!(h.read64(0x90).data, 0x135); + } + /// A failed BTE line stops the transfer and reports the failing source and destination. + #[test] + fn the_bte_reports_transfer_errors_and_stops_after_a_failed_line() { + { + for (read, fail) in [(true, 0x1080), (false, 0x2080)] { + let d = BedrockIo::new(); + let m = Arc::new(Memory::default()); + d.connect_memory(m.clone()); + if read { + *m.fail_read.lock() = Some(fail); + } else { + *m.fail_write.lock() = Some(fail); + } + launch(&d, reg::BTE1_BASE, 3, 0x1000, 0x2000, 0x3000, 0x10); + assert_eq!(d.read64(reg::BTE1_BASE + bte_reg::STATUS).data, 2 | ERROR); + assert_eq!(d.read64(reg::BTE1_BASE + bte_reg::SOURCE).data, 0x1080); + assert_eq!(d.read64(reg::BTE1_BASE + bte_reg::DESTINATION).data, 0x2080); + assert_eq!(m.read64(0x3000).data, 2 | ERROR); + } + let d = BedrockIo::new(); + let m = Arc::new(Memory::default()); + d.connect_memory(m.clone()); + *m.fail_write.lock() = Some(0x3000); + launch(&d, reg::BTE1_BASE, 1, 0x1000, 0x2000, 0x3000, 0x10); + assert_eq!(d.read64(reg::BTE1_BASE + bte_reg::STATUS).data, 0); + launch(&d, reg::BTE1_BASE, 2, 0xffffffff80, 0x2000, 0x3000, 0x10); + assert_eq!(d.read64(reg::BTE1_BASE + bte_reg::STATUS).data, 2 | ERROR); + } + { + let d = BedrockIo::new(); + launch(&d, reg::BTE1_BASE, 1, 0, 0, 0, 0x100); + assert_eq!(d.read64(reg::BTE1_BASE + bte_reg::STATUS).data, 1 | BUSY); + d.read64(reg::BTE1_BASE + bte_reg::CONTROL); + assert_eq!(d.read64(reg::BTE1_BASE + bte_reg::STATUS).data, 1); + launch(&d, reg::BTE1_BASE, 0, 0, 0, 0, 0x10); + assert_eq!(d.read64(reg::BTE1_BASE + bte_reg::STATUS).data, 0); + } + } + /// A downstream callback may reenter the IO block while an inbound transaction is active. + #[test] + fn reentrant_inbound_forwards_are_refused_and_can_be_retried() { + let d = Arc::new(BedrockIo::new()); + d.connect_memory(Arc::new(Reentrant(Arc::downgrade(&d)))); + assert_eq!(d.inbound_read(15, Some(0), 0, 8).status, BUS_ERR); + d.connect_memory(Arc::new(Busy)); + assert_eq!( + d.inbound_read(15, Some(0), 0, 8).status, + crate::traits::BUS_BUSY + ); + d.connect_widget(8, Arc::new(Busy)); + assert_eq!(d.pio_read(0x8000000, 8).status, crate::traits::BUS_BUSY); + assert_eq!(d.pio_write(0x8000000, 8, 42), crate::traits::BUS_BUSY); + + } + /// The two guest BTEs can reach node memory without one worker holding the other off. + #[test] + fn independent_bte_workers_can_reach_memory_concurrently() { + let d = Arc::new(BedrockIo::new()); + let (entered_tx, entered_rx) = std::sync::mpsc::channel(); + let (release_tx, release_rx) = std::sync::mpsc::channel(); + let memory = Arc::new(BlockingMemory { + entered: entered_tx, + release: Mutex::new(release_rx), + words: Mutex::new(BTreeMap::new()), + }); + d.connect_memory(memory.clone()); + let first_io = d.clone(); + let first = + std::thread::spawn(move || first_io.inbound_write(14, Some(0), 0x1000, 8, 0x1234)); + entered_rx + .recv_timeout(std::time::Duration::from_secs(5)) + .unwrap(); + let second_io = d.clone(); + let second = + std::thread::spawn(move || second_io.inbound_write(15, Some(1), 0x2000, 8, 0x5678)); + let second_status = second.join().unwrap(); + release_tx.send(()).unwrap(); + let first_status = first.join().unwrap(); + assert_eq!(second_status, BUS_OK); + assert_eq!(first_status, BUS_OK); + assert_eq!( + *memory.words.lock(), + BTreeMap::from([(0x1000, 0x1234), (0x2000, 0x5678)]) + ); + } +} diff --git a/src/dev/bedrock_lb.rs b/src/dev/bedrock_lb.rs new file mode 100644 index 000000000..7f3d86b3d --- /dev/null +++ b/src/dev/bedrock_lb.rs @@ -0,0 +1,516 @@ +//! Bedrock LB provides the node and region registers, atomic scratch aliases, +//! error and vector status, and the MicroLAN master used by board identity +//! parts. The guest polls the MicroLAN completion bit after issuing a slot. +//! +//! # What is approximated here +//! +//! The slot is decoded immediately and completes after one microsecond of +//! machine time. Permission registers retain guest writes, but this block's +//! access interface does not carry requester identity, so it cannot enforce +//! those permissions. +//! +//! The plain register words sit in a sparse map keyed by offset, since the +//! 0xc040-byte block holds only a few dozen of them. +//! +//! Ported from hw/misc/sgi-bedrock-lb.c in jamesbraid/qemu-sgi-origami. + +/// Length of the block-relative register aperture. +pub const SIZE: u64 = 0xc040; + +pub mod reg { + pub const NODE_REGION: u64 = 0x0; + pub const PIO_PERMISSION: u64 = 0x4000; + pub const PIO_PERMISSION_KEY: u64 = 0x4008; + pub const INBOUND_PERMISSION: u64 = 0x4010; + pub const SOFT_RESET: u64 = 0x4018; + pub const MICRO_LAN: u64 = 0x4030; + pub const ERROR_STATUS: u64 = 0x4040; + pub const ERROR_CLEAR: u64 = 0x4048; + pub const PIO_REGION_MASK: u64 = 0x4028; + pub const NODE_REGION_CONFIG: u64 = 0x4050; + pub const NODE_REGION_MASK: u64 = 0x4058; + pub const PIO_CONFIG: u64 = 0x4100; + pub const INBOUND_CONFIG: u64 = 0x4200; + pub const INBOUND_MASK: u64 = 0x4208; + pub const VECTOR_REQUEST: u64 = 0xc000; + pub const VECTOR_STATUS: u64 = 0xc020; + pub const VECTOR_STATUS_ALIAS: u64 = 0xc038; + pub const PIO_WINDOW: u64 = 0x8000; + pub const PIO_WINDOW_END: u64 = 0x8020; + pub const ATOMIC_SCRATCH: u64 = 0x8040; + pub const ATOMIC_SCRATCH_END: u64 = 0x8050; + pub const ATOMIC_READ0: u64 = 0x8058; + pub const ATOMIC_READ1: u64 = 0x8060; + pub const REGISTER_4020: u64 = 0x4020; + pub const REGISTER_4060: u64 = 0x4060; + pub const REGISTER_C008: u64 = 0xc008; + pub const REGISTER_C010: u64 = 0xc010; + pub const REGISTER_C028: u64 = 0xc028; + pub const REGISTER_C030: u64 = 0xc030; +} + +use crate::dev::bedrock_io::IIO_PROTECT_OVRRD_KEY; +use crate::dev::w1; +use crate::traits::{BusDevice, BusRead64, Resettable, BUS_ERR, BUS_OK}; +use parking_lot::{Mutex, ReentrantMutex}; +use std::collections::BTreeMap; +use std::sync::Arc; +pub type Clock = Arc u64 + Send + Sync>; +type Level = Arc; +const ID: u64 = (0x36 << 1) | (0x3012 << 12) | (2 << 28); +const VECTOR_MASK: u64 = 3 | (0x1fffff << 3) | (0xff << 32) | (0x7ff << 40); +struct Regs { + words: BTreeMap, + scratch: [u64; 5], + node: u8, + region: u8, + deadline: Option, + sample: bool, +} +impl Default for Regs { + fn default() -> Self { + let mut words = BTreeMap::new(); + words.insert(0x4000, !0); + words.insert(0x4010, !0); + Self { + words, + scratch: [0; 5], + node: 0, + region: 0, + deadline: None, + sample: true, + } + } +} +pub struct BedrockLb { + regs: Arc>, + clock: Clock, + error: Mutex>, + delivery: ReentrantMutex<()>, + driven: Mutex>, + reset: Mutex>>, + microlan: Mutex>, + node_changed: Mutex>>, +} +impl BedrockLb { + pub fn new(clock: Clock) -> Self { + let regs = Arc::new(Mutex::new(Regs::default())); + crate::locks::register_mutex("bedrock-lb", ®s); + Self { + regs, + clock, + error: Mutex::new(None), + delivery: ReentrantMutex::new(()), + driven: Mutex::new(None), + reset: Mutex::new(None), + microlan: Mutex::new(None), + node_changed: Mutex::new(None), + } + } + pub fn connect_error(&self, cb: Level) { + let _delivery = self.delivery.lock(); + *self.error.lock() = Some(cb); + *self.driven.lock() = None; + self.update_irq(); + } + /// Attach the node's 1-wire bus to the local-block MicroLAN master. + pub fn set_microlan(&self, bus: w1::Bus) { + *self.microlan.lock() = Some(bus); + } + pub fn connect_soft_reset(&self, cb: Arc) { + *self.reset.lock() = Some(cb); + } + pub fn connect_node_changed(&self, cb: Arc) { + let _delivery = self.delivery.lock(); + *self.node_changed.lock() = Some(cb); + } + pub fn node_number(&self) -> u8 { + self.regs.lock().node + } + pub fn error_level(&self) -> bool { + self.regs.lock().words.get(&0x4040).copied().unwrap_or(0) != 0 + } + fn update_irq(&self) { + // Serialize the sample and callback, while leaving registers available + // to the callback. Suppress unchanged levels so callbacks can reenter + // register reads that recompute the output. + let _delivery = self.delivery.lock(); + let cb = self.error.lock().clone(); + if let Some(cb) = cb { + let level = self.error_level(); + let changed = { + let mut driven = self.driven.lock(); + let changed = *driven != Some(level); + *driven = Some(level); + changed + }; + if changed { + cb(level); + } + } + } + fn notify_node(&self) { + let _delivery = self.delivery.lock(); + let cb = self.node_changed.lock().clone(); + if let Some(cb) = cb { + cb(self.node_number()); + } + } + fn complete(r: &mut Regs, now: u64) { + if r.deadline.is_some_and(|d| now >= d) { + *r.words.entry(0x4030).or_default() |= 2 | u64::from(r.sample); + r.deadline = None; + } + } + fn mask(a: u64) -> Option { + Some(match a { + reg::PIO_PERMISSION | reg::INBOUND_PERMISSION | reg::REGISTER_4020 | reg::REGISTER_4060 | reg::REGISTER_C008 | reg::REGISTER_C010 | reg::REGISTER_C028 | reg::REGISTER_C030 => !0, + reg::PIO_REGION_MASK => 0x80ff80ff80ff80ff, + reg::ERROR_STATUS => 0x7ff, + reg::NODE_REGION_CONFIG => 0x7f | (0x7ff << 8) | (0x7ff << 20) | (0xf << 32) | (1 << 40) | (1 << 44), + reg::NODE_REGION_MASK => (1 << 38) - 1, + reg::PIO_CONFIG => 0x3f, + reg::INBOUND_CONFIG => 1 | (0x3ff << 4) | (0x3ff << 16) | (1 << 28) | (1 << 32), + reg::INBOUND_MASK => { + 0x1f | (0xfff << 8) | (1 << 20) | (0x3ff << 24) | (0x3ff << 36) | (0x3ff << 48) + } + reg::VECTOR_REQUEST => VECTOR_MASK, + _ => return None, + }) + } +} +impl Resettable for BedrockLb { + fn power_on(&self) { + let mut r = self.regs.lock(); + let survivors = [ + r.scratch[1], + r.scratch[2], + r.words.get(®::ERROR_STATUS).copied().unwrap_or(0), + ]; + *r = Regs::default(); + r.scratch[1] = survivors[0]; + r.scratch[2] = survivors[1]; + r.words.insert(reg::ERROR_STATUS, survivors[2]); + drop(r); + self.update_irq(); + self.notify_node(); + } +} +impl BusDevice for BedrockLb { + fn read64(&self, a: u64) -> BusRead64 { + if a & 7 != 0 { + return BusRead64::err(); + } + let now = (self.clock)(); + let mut r = self.regs.lock(); + Self::complete(&mut r, now); + let v = match a { + reg::NODE_REGION => ID | ((r.node as u64) << 32) | ((r.region as u64) << 46), + reg::PIO_WINDOW..=reg::PIO_WINDOW_END => r.scratch[(a as usize - reg::PIO_WINDOW as usize) / 8], + reg::ATOMIC_READ0 | reg::ATOMIC_READ1 => { + let n = 3 + (a as usize - reg::ATOMIC_READ0 as usize) / 8; + let v = r.scratch[n]; + r.scratch[n] = 1; + v + } + reg::VECTOR_STATUS_ALIAS => r.words.remove(&0xc020).unwrap_or(0), + reg::VECTOR_STATUS | reg::MICRO_LAN => r.words.get(&a).copied().unwrap_or(0), + _ if Self::mask(a).is_some() => r.words.get(&a).copied().unwrap_or(0), + _ => { + crate::dlog!(crate::devlog::LogModule::Bedrock, "Bedrock LB unsupported read {a:#x}"); + 0 + } + }; + BusRead64::ok(v) + } + fn write64(&self, a: u64, v: u64) -> u32 { + if a & 7 != 0 { + return BUS_ERR; + } + let now = (self.clock)(); + let mut r = self.regs.lock(); + Self::complete(&mut r, now); + match a { + reg::NODE_REGION => { + let region = ((v >> 46) & 3) as u8; + if region == 3 { + crate::dlog!(crate::devlog::LogModule::Bedrock, "Bedrock LB invalid region size"); + } else { + r.node = (v >> 32) as u8; + r.region = region; + } + } + reg::PIO_WINDOW..=reg::PIO_WINDOW_END => { + let n = (a as usize - 0x8000) / 8; + r.scratch[n] = if n < 3 { v } else { v & 1 }; + } + reg::ATOMIC_SCRATCH..=reg::ATOMIC_SCRATCH_END => { + let n = (a as usize - reg::ATOMIC_SCRATCH as usize) / 8; + if r.scratch[n] == 0 { + r.scratch[n] = v; + } + } + reg::PIO_PERMISSION_KEY => { + if v == IIO_PROTECT_OVRRD_KEY { + *r.words.entry(reg::PIO_PERMISSION).or_default() |= 1; + } + } + reg::SOFT_RESET => {} + reg::MICRO_LAN => { + if r.deadline.is_some() { + crate::dlog!(crate::devlog::LogModule::Bedrock, "Bedrock LB MicroLAN write while slot pending discarded"); + } else { + r.words.insert(a, v & ((1 << 34) - 4)); + let (pulse, sample) = w1::slot::fields(v); + r.sample = match self.microlan.lock().as_mut() { + Some(bus) => w1::slot::slot(bus, pulse, sample), + None => pulse >= w1::slot::RESET_PULSE + || pulse < w1::slot::ONE_PULSE, + }; + r.deadline = Some(now.saturating_add(1000)); + } + } + reg::ERROR_STATUS => { + *r.words.entry(a).or_default() |= v & 0x7ff; + } + reg::ERROR_CLEAR => { + *r.words.entry(0x4040).or_default() &= !(v & 0x7ff); + } + reg::VECTOR_REQUEST => { + r.words.insert(a, v & VECTOR_MASK); + if r.words.get(®::VECTOR_STATUS).copied().unwrap_or(0) & (1 << 63) != 0 { + *r.words.entry(reg::VECTOR_STATUS).or_default() |= 1 << 62; + } else { + crate::dlog!(crate::devlog::LogModule::Bedrock, "Bedrock LB vector launch reaches no peer; no reply latched"); + } + } + reg::VECTOR_STATUS | reg::VECTOR_STATUS_ALIAS => { + if v == 0 { + r.words.insert(reg::VECTOR_STATUS, 0); + } else { + crate::dlog!(crate::devlog::LogModule::Bedrock, "Bedrock LB nonzero vector status write ignored"); + } + } + _ => { + if let Some(mask) = Self::mask(a) { + r.words.insert(a, v & mask); + if (a == reg::PIO_PERMISSION || a == reg::INBOUND_PERMISSION) && v != !0 { + crate::dlog!(crate::devlog::LogModule::Bedrock, "Bedrock LB permission enforcement requires requester context"); + } + } else { + crate::dlog!(crate::devlog::LogModule::Bedrock, "Bedrock LB unsupported write {a:#x} = {v:#x}"); + } + } + } + drop(r); + self.update_irq(); + if a == reg::NODE_REGION { + self.notify_node(); + } + if a == reg::SOFT_RESET && v & 1 != 0 { + self.power_on(); + let cb = self.reset.lock().clone(); + if let Some(cb) = cb { + cb(); + } + } + BUS_OK + } +} + +#[cfg(test)] +mod tests { + // Cases follow tests/qtest/sn1-bedrock-lb-test.c in + // jamesbraid/qemu-sgi-origami: identity-permissions, scratch-reset and + // microlan-slot. + use super::*; + use std::sync::atomic::{AtomicBool, AtomicU8, AtomicU64, Ordering}; + /// Hold an IRQ packet's assertion callback while a CPU clears the register. + pub(super) fn check_order( + device: Arc, + install: impl FnOnce(Level), + raise: u64, + value: u64, + clear: impl FnOnce() + Send + 'static, + ) { + let (entered_tx, entered_rx) = std::sync::mpsc::channel(); + let (release_tx, release_rx) = std::sync::mpsc::channel(); + let release_rx = Mutex::new(release_rx); + let delivered = Arc::new(AtomicBool::new(false)); + let sink = delivered.clone(); + install(Arc::new(move |level| { + if level { + entered_tx.send(()).unwrap(); + release_rx + .lock() + .recv_timeout(std::time::Duration::from_secs(5)) + .unwrap(); + } + sink.store(level, Ordering::SeqCst); + })); + let io = crate::dev::bedrock_io::BedrockIo::new(); + io.connect_hub(device); + let first = std::thread::spawn(move || { + io.inbound_write(14, Some(0), (1 << 47) | 0x1800000 | raise, 8, value) + }); + entered_rx + .recv_timeout(std::time::Duration::from_secs(5)) + .unwrap(); + let (done_tx, done_rx) = std::sync::mpsc::channel(); + let second = std::thread::spawn(move || { + clear(); + done_tx.send(()).unwrap(); + }); + // An ordered callback waits for the blocked assertion. A stale publisher + // instead completes the clear early, then overwrites it after release. + let _ = done_rx.recv_timeout(std::time::Duration::from_millis(100)); + release_tx.send(()).unwrap(); + assert_eq!(first.join().unwrap(), BUS_OK); + second.join().unwrap(); + assert!(!delivered.load(Ordering::SeqCst)); + } + + /// The guest configures node permissions and expects selected scratch words to survive reset. + #[test] + fn permission_writes_latch_errors_and_reset_retains_scratch() { + { + let d = BedrockLb::new(Arc::new(|| 0)); + d.write64(reg::NODE_REGION, (0x5a << 32) | (2 << 46)); + assert_eq!(d.node_number(), 0x5a); + d.write64(reg::NODE_REGION, 3 << 46); + assert_eq!(d.node_number(), 0x5a); + d.write64(reg::PIO_PERMISSION, 0); + d.write64(reg::PIO_PERMISSION_KEY, 0); + assert_eq!(d.read64(reg::PIO_PERMISSION).data, 0); + d.write64(reg::PIO_PERMISSION_KEY, IIO_PROTECT_OVRRD_KEY); + assert_eq!(d.read64(reg::PIO_PERMISSION).data, 1); + d.write64(reg::PIO_WINDOW + 8, 42); + d.write64(reg::ERROR_STATUS, !0); + assert_eq!(d.read64(reg::ERROR_STATUS).data, 0x7ff); + d.power_on(); + assert_eq!(d.read64(reg::PIO_WINDOW + 8).data, 42); + assert!(d.error_level()); + d.write64(reg::ERROR_CLEAR, 0x7ff); + assert!(!d.error_level()); + assert_eq!(d.node_number(), 0); + } + { + let d = BedrockLb::new(Arc::new(|| 0)); + d.write64(reg::ATOMIC_SCRATCH, 42); + d.write64(reg::ATOMIC_SCRATCH, 43); + assert_eq!(d.read64(reg::PIO_WINDOW).data, 42); + d.write64(reg::REGISTER_C030, 0xdead); + d.write64(reg::VECTOR_REQUEST, !0); + assert_eq!(d.read64(reg::VECTOR_STATUS).data, 0); + assert_eq!(d.read64(reg::REGISTER_C030).data, 0xdead); + assert_eq!(d.read32(reg::PIO_WINDOW).status, BUS_ERR); + assert_eq!(d.read64(1).status, BUS_ERR); + } + } + /// The guest polls MicroLAN completion and cannot replace a slot that is still pending. + #[test] + fn microlan_slots_finish_after_one_microsecond_and_reject_overlap() { + let time = Arc::new(AtomicU64::new(0)); + let t = time.clone(); + let d = BedrockLb::new(Arc::new(move || t.load(Ordering::SeqCst))); + let word = (0x66 << 27) | (60 << 10) | (0x41 << 2) | 3; + d.write64(reg::MICRO_LAN, word); + assert_eq!(d.read64(reg::MICRO_LAN).data, word & !3); + d.write64(reg::MICRO_LAN, 8 << 10); + assert_eq!(d.read64(reg::MICRO_LAN).data, word & !3); + time.store(1000, Ordering::SeqCst); + assert_eq!(d.read64(reg::MICRO_LAN).data, (word & !3) | 2); + d.write64(reg::MICRO_LAN, 8 << 10); + time.store(2000, Ordering::SeqCst); + assert_eq!(d.read64(reg::MICRO_LAN).data, (8 << 10) | 3); + d.write64(reg::MICRO_LAN, 60 << 10); + d.power_on(); + time.store(3000, Ordering::SeqCst); + assert_eq!(d.read64(reg::MICRO_LAN).data, 0); + } + /// Node-change callbacks may reenter LB while guest writes race. + #[test] + fn notifications_preserve_concurrent_register_order() { + { + let ni = Arc::new(crate::dev::bedrock_ni::BedrockNi::new()); + let a = ni.clone(); + let b = ni.clone(); + check_order( + ni, + move |cb| a.connect_error(cb), + crate::dev::bedrock_ni::reg::ERROR_STATUS, + 1 << 24, + move || { + b.read64(crate::dev::bedrock_ni::reg::ERROR_CLEAR); + }, + ); + } + { + let d = Arc::new(BedrockLb::new(Arc::new(|| 0))); + let a = d.clone(); + let b = d.clone(); + check_order( + d, + move |cb| a.connect_error(cb), + reg::ERROR_STATUS, + 1, + move || { + b.write64(reg::ERROR_CLEAR, 1); + }, + ); + } + { + let d = Arc::new(crate::dev::bedrock_xb::BedrockXb::new()); + let a = d.clone(); + let b = d.clone(); + check_order( + d, + move |cb| a.connect_error(cb), + crate::dev::bedrock_xb::reg::QUEUE_BASE, + 1, + move || { + b.write64(crate::dev::bedrock_xb::reg::CLEAR_QUEUE_BASE, 0); + b.write64(crate::dev::bedrock_xb::reg::CLEAR_FIRST_ERROR, 0); + }, + ); + } + { + let d = Arc::new(BedrockLb::new(Arc::new(|| 0))); + let delivered = Arc::new(AtomicU8::new(0)); + let sink = delivered.clone(); + let (entered_tx, entered_rx) = std::sync::mpsc::channel(); + let (release_tx, release_rx) = std::sync::mpsc::channel(); + let release_rx = Mutex::new(release_rx); + d.connect_node_changed(Arc::new(move |node| { + if node == 1 { + entered_tx.send(()).unwrap(); + release_rx + .lock() + .recv_timeout(std::time::Duration::from_secs(5)) + .unwrap(); + } + sink.store(node, Ordering::SeqCst); + })); + let io = crate::dev::bedrock_io::BedrockIo::new(); + io.connect_hub(d.clone()); + let first = std::thread::spawn(move || { + io.inbound_write(14, Some(0), (1 << 47) | 0x1800000, 8, 1 << 32) + }); + entered_rx + .recv_timeout(std::time::Duration::from_secs(5)) + .unwrap(); + let (done_tx, done_rx) = std::sync::mpsc::channel(); + let other = d.clone(); + let second = std::thread::spawn(move || { + other.write64(0, 2 << 32); + done_tx.send(()).unwrap(); + }); + let _ = done_rx.recv_timeout(std::time::Duration::from_millis(100)); + release_tx.send(()).unwrap(); + assert_eq!(first.join().unwrap(), BUS_OK); + second.join().unwrap(); + assert_eq!(d.node_number(), 2); + assert_eq!(delivered.load(Ordering::SeqCst), 2); + } + } +} diff --git a/src/dev/bedrock_md.rs b/src/dev/bedrock_md.rs new file mode 100644 index 000000000..f3e41f012 --- /dev/null +++ b/src/dev/bedrock_md.rs @@ -0,0 +1,1549 @@ +//! Bedrock's memory and directory controller: bank configuration, memory +//! self-test, ECC, scrubbing and the directory and fetch-operation views. +//! +//! Memory is held by `bedrock_mem::BedrockMemory`. CPU coherence requests +//! and hardware protection enforcement are absent; protection registers +//! retain their words. Known unsupported writes are counted and ignored, +//! and unmapped accesses return a bus error. +//! +//! Plain register words, directory entries and reference counters, +//! protection entries and injected ECC check bytes live in sparse maps. The +//! guest touches a few of the millions of possible entries, and an entry it +//! never wrote reads as its default, so a dense table would only cost memory. +//! +//! Ported from hw/misc/sgi-bedrock-md.c in jamesbraid/qemu-sgi-origami. + +use super::bedrock_mem::{BedrockMemory, BANK_SPAN, FITTED_BANK}; +use crate::traits::*; +use parking_lot::Mutex; +use std::collections::{BTreeSet, HashMap, HashSet}; +use std::sync::atomic::{AtomicU64, Ordering}; +use std::sync::{Arc, Weak}; + + +pub mod reg { + pub const CELL: u64 = 0x0; + pub const CONFIG: u64 = 0x8; + pub const ARBITRATION: u64 = 0x10; + pub const MIGRATION_CONFIG: u64 = 0x18; + /// Outgoing reply queue sizes: read-only, so a write changes nothing. + pub const REPLY_QUEUE_SIZE: u64 = 0x60; + pub const PERF_SELECT0: u64 = 0x10000; + pub const PERF_SELECT1: u64 = 0x10008; + pub const PERF_COUNTER0: u64 = 0x10010; + pub const PERF_COUNTER1: u64 = 0x10018; + pub const REFRESH: u64 = 0x20000; + /// Flash, UART and Synergy timing on the local "junk" bus. + pub const JUNK_BUS_TIMING: u64 = 0x20008; + pub const SELFTEST_COMMAND: u64 = 0x20030; + pub const SELFTEST_DATA: u64 = 0x20038; + pub const SELFTEST_ERROR: u64 = 0x20040; + pub const SELFTEST_STATUS: u64 = 0x20048; + pub const ISSUE_DEBUG: u64 = 0x20060; + pub const DIRECTORY_CONFIG: u64 = 0x40000; + pub const DIRECTORY_ERROR: u64 = 0x40010; + /// Reads the directory error record and clears it. + pub const DIRECTORY_ERROR_ALIAS: u64 = 0x40018; + pub const MIGRATION_DIFFERENCE: u64 = 0x40040; + pub const MIGRATION_VALUE: u64 = 0x40048; + /// Outgoing request queue sizes; the model keeps them fixed. + pub const REQUEST_QUEUE_SIZE: u64 = 0x40050; + pub const DIRECTORY_DEBUG: u64 = 0x40060; + pub const ECC_CONFIG: u64 = 0x60000; + pub const MEMORY_ERROR: u64 = 0x60010; + /// Reads the memory error record and clears it. + pub const MEMORY_ERROR_ALIAS: u64 = 0x60018; + pub const MEMORY_DEBUG: u64 = 0x60040; + + /// Reset sizes: 12 entries in each header field and 24 in each data field. + pub const REPLY_QUEUE_SIZES: u64 = 0x0018_1818_0c0c_0c00; + /// Reset sizes: four requests for each of the three queues. + pub const REQUEST_QUEUE_SIZES: u64 = 0x0404_0400; +} + +const CONFIG_MASK: u64 = 0xfbfbfbfb | (3 << 32) | (15 << 44) | (63 << 48) | (7 << 56) | (3 << 60); +const ECC_COLUMNS: [u8; 72] = [ + 0x13, 0x23, 0x43, 0x83, 0x2f, 0xf1, 0x0d, 0x07, 0xd0, 0x70, 0x4f, 0xf8, 0x61, 0x62, 0x64, 0x68, + 0x1c, 0x2c, 0x4c, 0x8c, 0x15, 0x25, 0x45, 0x85, 0x19, 0x29, 0x49, 0x89, 0x1a, 0x2a, 0x4a, 0x8a, + 0x51, 0x52, 0x54, 0x58, 0x91, 0x92, 0x94, 0x98, 0xa1, 0xa2, 0xa4, 0xa8, 0x31, 0x32, 0x34, 0x38, + 0x16, 0x26, 0x46, 0x86, 0x1f, 0xf2, 0x0b, 0x0e, 0xb0, 0xe0, 0x8f, 0xf4, 0xc1, 0xc2, 0xc4, 0xc8, + 1, 2, 4, 8, 16, 32, 64, 128, +]; +pub fn ecc_compute(mut value: u64) -> u8 { + let mut code = 0; + while value != 0 { + code ^= ECC_COLUMNS[value.trailing_zeros() as usize]; + value &= value - 1; + } + code +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum MemoryError { + Correctable, + Uncorrectable, +} +type ErrorCallback = Arc; + +struct State { + cell: u64, + config: u64, + refresh: u64, + directory_config: u64, + ecc_config: u64, + storage: HashMap, + directory: HashMap, + counters: HashMap, + protection: HashMap, + compat: HashMap, + ecc: HashMap, + admitted: [u16; 2], + memory_error: u64, + directory_error: u64, + marked_directory: HashSet, + clock_ns: u64, + clock_anchor: Option, + selftest_deadline: Option, + selftest_result: bool, + selftest_status: u64, + selftest_error: u64, +} +impl Default for State { + fn default() -> Self { + Self { + cell: u64::MAX, + config: 0, + refresh: 0, + directory_config: 0, + ecc_config: 0, + storage: HashMap::new(), + directory: HashMap::new(), + counters: HashMap::new(), + protection: HashMap::new(), + compat: HashMap::new(), + ecc: HashMap::new(), + admitted: [0; 2], + memory_error: 0, + directory_error: 0, + marked_directory: HashSet::new(), + clock_ns: 0, + clock_anchor: None, + selftest_deadline: None, + selftest_result: false, + selftest_status: 0, + selftest_error: 0, + } + } +} + +pub struct BedrockMd { + memory: BedrockMemory, + clock: Option, + // Ordinary RAM uses Physical's bank views. Diagnostic metadata, repair + // and fetch operations retain the state lock; direct bank traffic has + // the same concurrency contract as the other machines' RAM. + state: Mutex, + // CPU-side metadata writes and resets serialize snapshot/publication. + // RAM/DMA reads and writes may remove metadata without taking this lock. + remapping: Mutex<()>, + callback: Mutex>, + unsupported: AtomicU64, +} +#[derive(Clone, Copy)] +enum View { + Registers, + Memory, + Uncached, + Scrub, + Directory, + Fetchop, + Ecc, +} +struct MdBus { + md: Weak, + view: View, + // External endpoints may outlive MD. Keep their generation pointers valid. + // The endpoint owned by Physical itself needs no extra reference. + _memory: Option>, + #[cfg(feature = "jitv2")] + _generations: Arc, +} + +impl BedrockMd { + pub fn new() -> Arc { + Self::build(None) + } + pub fn with_clock(clock: crate::dev::heart::Clock) -> Arc { + Self::build(Some(clock)) + } + fn build(clock: Option) -> Arc { + let md = Arc::new(Self { + memory: BedrockMemory::new(), + clock, + state: Mutex::new(State::default()), + remapping: Mutex::new(()), + callback: Mutex::new(None), + unsupported: AtomicU64::new(0), + }); + md.memory.set_slow_view(md.view(View::Memory)); + md.rebuild_memory(); + md + } + pub fn register_locks(self: &Arc) { + let weak = Arc::downgrade(self); + crate::locks::register_lock_fn("bedrock-md::state", move || { + weak.upgrade().is_some_and(|md| md.state.is_locked()) + }); + let weak = Arc::downgrade(self); + crate::locks::register_lock_fn("bedrock-md::remapping", move || { + weak.upgrade().is_some_and(|md| md.remapping.is_locked()) + }); + let weak = Arc::downgrade(self); + crate::locks::register_lock_fn("bedrock-md::callback", move || { + weak.upgrade().is_some_and(|md| md.callback.is_locked()) + }); + } + pub fn physical(&self) -> &Arc { + self.memory.physical() + } + fn view(self: &Arc, view: View) -> Arc { + Arc::new(MdBus { + md: Arc::downgrade(self), + view, + #[cfg(feature = "jitv2")] + _generations: self.memory.generations(), + _memory: if matches!(view, View::Uncached) { + Some(self.physical().clone()) + } else { + None + }, + }) + } + pub fn md_bus(self: &Arc) -> Arc { + self.view(View::Registers) + } + pub fn directory_bus(self: &Arc) -> Arc { + self.view(View::Directory) + } + pub fn fetchop_bus(self: &Arc) -> Arc { + self.view(View::Fetchop) + } + pub fn ecc_bus(self: &Arc) -> Arc { + self.view(View::Ecc) + } + /// DMA uses the same Physical entries as the CPU, without accessing + /// ppmem's CPU-owned bitmap. Hardware protection remains unsupported. + pub fn dma_view(self: &Arc) -> Arc { + Arc::new(DmaView(self.clone())) + } + + fn mapping(s: &State) -> (u64, BTreeSet) { + let pages = s.ecc.keys().map(|key| key / (4096 / 8)) + .chain(s.marked_directory.iter().map(|key| key / (4096 / 128))) + .collect(); + (s.config, pages) + } + + /// Reinstall RAM after the board sets Physical's default. Call on the + /// CPU thread or with the CPU stopped, before installing board overlays. + /// Ordinary CPU config/diagnostic writes rebuild their changed mappings + /// automatically. This also releases pages whose metadata DMA consumed. + pub fn rebuild_memory(&self) { + let _remapping = self.remapping.lock(); + let (config, pages) = Self::mapping(&self.state.lock()); + self.memory.remap(config, pages, true); + } + /// The stable logical-physical counters used by both compiled stores + /// and SysAd page claims. Their storage outlives the CPU via its bus. + #[cfg(feature = "jitv2")] + pub(crate) fn generation_base(&self) -> *mut AtomicU64 { + self.memory.generations().base() + } + pub fn on_error(&self, callback: ErrorCallback) { + *self.callback.lock() = Some(callback); + } + /// Explicit elapsed time for a manually clocked controller. + pub fn advance_ns(&self, elapsed: u64) { + let mut s = self.state.lock(); + Self::advance(&mut s, elapsed); + } + fn advance(s: &mut State, elapsed: u64) { + s.clock_ns = s.clock_ns.saturating_add(elapsed); + if s.selftest_deadline.is_some_and(|end| s.clock_ns >= end) { + s.selftest_deadline = None; + s.selftest_status = if s.selftest_result { 3 } else { 2 }; + s.selftest_error = if s.selftest_result { 0 } else { 1 << 48 }; + } + } + fn catch_up(s: &mut State, now: u64) { + if let Some(anchor) = s.clock_anchor { + s.clock_anchor = Some(anchor.max(now)); + Self::advance(s, now.saturating_sub(anchor)); + } + } + pub fn start_clock(&self) { + if let Some(clock) = &self.clock { + let now = clock(); + self.state.lock().clock_anchor.get_or_insert(now); + } + } + pub fn stop_clock(&self) { + if let Some(clock) = &self.clock { + let now = clock(); + let mut s = self.state.lock(); + Self::catch_up(&mut s, now); + s.clock_anchor = None; + } + } + /// Normal MD memory semantics without re-entering the cached Physical map. + pub fn memory_view(self: &Arc) -> Arc { + self.view(View::Memory) + } + pub fn uncached_view(self: &Arc) -> Arc { + self.view(View::Uncached) + } + pub fn ecc_scrub_bus(self: &Arc) -> Arc { + self.view(View::Scrub) + } + pub fn current_cell(&self) -> u64 { + self.state.lock().cell + } + pub fn unsupported(&self) -> u64 { + self.unsupported.load(Ordering::Relaxed) + } + /// Count and log an access the model does not implement; the caller + /// answers a bus error. + fn refuse(&self, what: &str, at: u64) -> Option { + self.unsupported.fetch_add(1, Ordering::Relaxed); + crate::dlog!(crate::devlog::LogModule::Bedrock, + "Bedrock MD refused {what} access at {at:#x}"); + None + } + /// Called on the CPU thread or with the CPU stopped. + pub fn soft_reset(&self) { + let _remapping = self.remapping.lock(); + let mut s = self.state.lock(); + s.cell = u64::MAX; + s.storage.clear(); + s.ecc_config = 0; + s.memory_error = 0; + s.directory_error = 0; + s.ecc.clear(); + s.marked_directory.clear(); + s.selftest_deadline = None; + s.selftest_status = 0; + s.selftest_error = 0; + s.refresh &= (1 << 63) | 0xfff; + let (config, pages) = Self::mapping(&s); + drop(s); + self.memory.remap(config, pages, false); + } + fn access(&self, view: View, addr: u64, bytes: u64, write: Option) -> Option { + if ![1, 2, 4, 8].contains(&bytes) || addr % bytes != 0 { + return self.refuse("access", addr); + } + // Only CPU register/configuration and diagnostic writes can add + // metadata. Keep their remap off DMA threads: ordinary RAM, scrub + // and fetch operations only remove entries, leaving safe slow pages + // behind until a CPU-side rebuild. These write views, like HEART's + // MEMCFG, are CPU-thread-only while the machine runs. + let publish = write.is_some() && match view { + View::Registers => matches!(addr, reg::CONFIG | reg::SELFTEST_COMMAND), + View::Directory | View::Ecc => true, + _ => false, + }; + let remapping = publish.then(|| self.remapping.lock()); + let mut event = Vec::new(); + // Sample outside state: even an injected clock must not call out under + // the memory lock. Only register observations need the self-test timer. + let now = if matches!(view, View::Registers) { + self.clock.as_ref().map(|clock| clock()) + } else { + None + }; + let (result, snapshot) = { + let mut s = self.state.lock(); + if let Some(now) = now { + Self::catch_up(&mut s, now); + } + let result = match view { + View::Registers => self.register(&mut s, addr, bytes, write), + View::Directory => self.directory(&mut s, addr, bytes, write), + View::Ecc => self.ecc(&mut s, addr, bytes, write), + View::Scrub => self.scrub(&mut s, addr, bytes, write, &mut event), + View::Uncached => { + let result = self.ram(&mut s, addr, bytes, write, &mut event); + if write.is_none() && event.contains(&MemoryError::Uncorrectable) { + None + } else { + result + } + } + View::Fetchop => self.fetchop(&mut s, addr, bytes, write, &mut event), + View::Memory => self.ram(&mut s, addr, bytes, write, &mut event), + }; + (result, publish.then(|| Self::mapping(&s))) + }; + if let Some((config, pages)) = snapshot { + self.memory.remap(config, pages, false); + } + drop(remapping); + if !event.is_empty() { + let callback = self.callback.lock().clone(); + if let Some(callback) = callback { + for kind in event { + callback(kind); + } + } + } + result + } + fn register(&self, s: &mut State, at: u64, bytes: u64, write: Option) -> Option { + if bytes != 8 { + return self.refuse("register", at); + } + let old = match at { + reg::CELL => s.cell, + reg::CONFIG => s.config, + reg::REPLY_QUEUE_SIZE => reg::REPLY_QUEUE_SIZES, + reg::REFRESH => s.refresh, + reg::SELFTEST_ERROR => s.selftest_error, + reg::SELFTEST_STATUS => s.selftest_status, + reg::DIRECTORY_CONFIG => s.directory_config, + reg::ECC_CONFIG => s.ecc_config, + reg::REQUEST_QUEUE_SIZE => reg::REQUEST_QUEUE_SIZES, + reg::MEMORY_ERROR | reg::MEMORY_ERROR_ALIAS => s.memory_error, + reg::DIRECTORY_ERROR | reg::DIRECTORY_ERROR_ALIAS => s.directory_error, + _ if storage_mask(at).is_some() => *s.storage.get(&at).unwrap_or(&0), + _ if idle(at) => 0, + _ => return self.refuse("register", at), + }; + if let Some(value) = write { + match at { + reg::SELFTEST_COMMAND => return self.selftest(s, value), + reg::CELL => s.cell = value, + reg::CONFIG => { + s.config = value & CONFIG_MASK; + } + reg::REFRESH => s.refresh = value & ((1 << 63) | 0xfff), + reg::DIRECTORY_CONFIG => s.directory_config = value & 3, + reg::ECC_CONFIG => s.ecc_config = value & 1, + _ if storage_mask(at).is_some() => { + s.storage.insert(at, value & storage_mask(at).unwrap()); + } + _ => { + self.unsupported.fetch_add(1, Ordering::Relaxed); + crate::dlog!(crate::devlog::LogModule::Bedrock, + "Bedrock MD ignored register write at {at:#x}: {value:#x}"); + } + } + } else if at == reg::MEMORY_ERROR_ALIAS { + s.memory_error = 0; + } else if at == reg::DIRECTORY_ERROR_ALIAS { + s.directory_error = 0; + } + Some(old) + } + fn directory(&self, s: &mut State, at: u64, bytes: u64, write: Option) -> Option { + if bytes != 8 || at >= 1 << 30 { + return self.refuse("directory", at); + } + let software = at / (BANK_SPAN / 8); + let bank = software ^ ((s.config >> 32) & 3); + if at & 0x7c8 == 0x408 { + if bank >= 2 { + return Some(0); + } + let within = + ((((at % (BANK_SPAN / 8)) >> 11) << 14) | (((at >> 4) & 3) << 7)) % FITTED_BANK; + let key = ((bank * FITTED_BANK + within) >> 14) * 4 + ((within >> 7) & 3); + let old = *s.protection.get(&key).unwrap_or(&0); + if let Some(v) = write { + s.protection.insert(key, v); + } + return Some(old); + } + let within = ((at % (BANK_SPAN / 8)) & !15) * 8; + let field = s.config >> (bank / 2 * 8); + let code = (field >> 4) & 15; + if field & (1 << (bank & 1)) == 0 + || !(1..=5).contains(&code) + || within >= (32 << 20) << code + { + return self.refuse("directory", at); + } + let mask = if at & 8 != 0 || s.directory_config & 1 != 0 { + u64::MAX + } else { + u32::MAX as u64 + }; + let table; + let key; + if bank >= 2 || within >= FITTED_BANK { + if !s.compat.contains_key(&at) && s.compat.len() >= 74 { + return self.refuse("directory", at); + } + s.compat.entry(at).or_insert(0); + table = &mut s.compat; + key = at; + } else { + key = (bank * FITTED_BANK + within) / 128; + table = if at & 8 != 0 { + &mut s.counters + } else { + &mut s.directory + }; + } + let old = *table.get(&key).unwrap_or(&0) & mask; + if let Some(value) = write { + let value = if at & 8 == 0 && s.directory_config & 2 == 0 { + directory_encode(value, s.directory_config & 1 != 0) + } else { + value + }; + table.insert(key, value & mask); + if bank < 2 && within < FITTED_BANK && at & 8 == 0 { + if directory_syndrome(value, s.directory_config & 1 != 0) != 0 { + s.marked_directory.insert(key); + } else { + s.marked_directory.remove(&key); + } + } + } + Some(old) + } + fn checked_word( + &self, + s: &mut State, + bank: usize, + at: u64, + address: u64, + event: &mut Vec, + ) -> u64 { + let mut data = self.memory.raw_read(bank, at); + if s.ecc_config & 1 != 0 || s.ecc.is_empty() { + return data; + } + let key = (bank as u64 * FITTED_BANK + at) / 8; + if let Some(&check) = s.ecc.get(&key) { + let syndrome = check ^ ecc_compute(data); + if syndrome != 0 { + let bit = ECC_COLUMNS.iter().position(|&v| v == syndrome); + let valid = if bit.is_some() { 1 << 52 } else { 1 << 60 }; + if bit.is_none() + && s.memory_error & (1 << 60) != 0 + && (s.memory_error ^ address) & (((1u64 << 30) - 1) << 3) & !127 != 0 + { + s.memory_error |= 1 << 63; + } + if s.memory_error & (valid | (1 << 60)) != 0 { + s.memory_error |= valid << 1; + } else { + s.memory_error = (s.memory_error + & !((0xff << 40) | (((1u64 << 30) - 1) << 3))) + | ((syndrome as u64) << 40) + | (address & (((1u64 << 30) - 1) << 3)); + } + s.memory_error |= valid; + if let Some(bit) = bit { + if bit < 64 { + data ^= 1 << bit; + self.memory.raw_write(bank, at, data); + } + s.ecc.remove(&key); + event.push(MemoryError::Correctable); + } else { + event.push(MemoryError::Uncorrectable); + } + } + } + data + } + fn ram( + &self, + s: &mut State, + address: u64, + bytes: u64, + write: Option, + event: &mut Vec, + ) -> Option { + let Some((bank, offset)) = BedrockMemory::decode(s.config, address, bytes) else { + return self.refuse("ram", address); + }; + if !s.marked_directory.is_empty() { + let line = (bank as u64 * FITTED_BANK + offset) / 128; + if s.marked_directory.contains(&line) { + let premium = s.directory_config & 1 != 0; + let word = *s.directory.get(&line).unwrap_or(&0); + if write.is_some() { + s.directory.insert(line, directory_encode(word, premium)); + s.marked_directory.remove(&line); + } else if s.directory_config & 2 == 0 { + let syndrome = directory_syndrome(word, premium); + if syndrome != 0 { + let bit = directory_columns(premium) + .iter() + .position(|&v| v == syndrome); + let valid = if bit.is_some() { 1 << 61 } else { 1 << 63 }; + if s.directory_error & ((1 << 63) | valid) != 0 { + s.directory_error |= if bit.is_some() { 1 << 57 } else { 1 << 59 }; + } else { + let carrier = 0x1c0000000 + + (address / BANK_SPAN) * (BANK_SPAN / 8) + + ((offset & !127) / 8); + s.directory_error = (s.directory_error + & !((0x7f << 40) | (((1u64 << 30) - 1) << 3))) + | ((syndrome as u64) << 40) + | (carrier & (((1u64 << 30) - 1) << 3)); + } + s.directory_error |= valid; + if let Some(bit) = bit { + s.directory.insert(line, word ^ (1 << bit)); + s.marked_directory.remove(&line); + event.push(MemoryError::Correctable); + } else { + event.push(MemoryError::Uncorrectable); + return None; + } + } + } + } + } + let at = offset & !7; + let shift = (8 - (offset & 7) - bytes) * 8; + let mask = u64::MAX >> ((8 - bytes) * 8); + if let Some(value) = write { + let old = self.memory.raw_read(bank, at); + self.memory.raw_write( + bank, + at, + (old & !(mask << shift)) | ((value & mask) << shift), + ); + if !s.ecc.is_empty() { + s.ecc.remove(&((bank as u64 * FITTED_BANK + at) / 8)); + } + Some(0) + } else { + Some((self.checked_word(s, bank, at, address, event) >> shift) & mask) + } + } + fn fetchop( + &self, + s: &mut State, + at: u64, + bytes: u64, + write: Option, + event: &mut Vec, + ) -> Option { + if bytes != 4 && bytes != 8 { + return self.refuse("fetchop", at); + } + let op = at & 0x38; + if op > if write.is_some() { 32 } else { 24 } { + return self.refuse("fetchop", at); + } + let address = (at & !63) | (at & 4); + let old = self.ram(s, address, bytes, None, event)?; + let next = match (write, op) { + (None, 0) => return Some(old), + (Some(v), 0) => v, + (_, 8) => old.wrapping_add(1), + (_, 16) => old.wrapping_sub(1), + (None, 24) => 0, + (Some(v), 24) => old & v, + (Some(v), 32) => old | v, + _ => return self.refuse("fetchop", at), + }; + self.ram(s, address, bytes, Some(next), event)?; + Some(old) + } + // Approximation: report whether the selected fitting covers the + // programmed bank, after 1000 guest ns. Only the zero write phase mutates + // memory; other patterns have no published transformation here. + fn selftest(&self, s: &mut State, command: u64) -> Option { + if command == 2 || command == 4 { + s.selftest_deadline = None; + s.selftest_status = 0; + s.selftest_error = 0; + return Some(0); + } + if command & 7 != 1 || command & !0x71 != 0 { + self.unsupported.fetch_add(1, Ordering::Relaxed); + crate::dlog!(crate::devlog::LogModule::Bedrock, + "Bedrock MD ignored self-test command {command:#x}"); + return Some(0); + } + let selected = (command >> 4) & 7; + let configured = selected ^ ((s.config >> 32) & 3); + let field = s.config >> ((configured / 2) * 8); + let code = (field >> 4) & 15; + let size = if (1..=5).contains(&code) { + (32 << 20) << code + } else { + 0 + }; + let fits = selected < 2 + && field & (1 << (configured & 1)) != 0 + && size != 0 + && size <= FITTED_BANK; + s.selftest_result = fits; + s.selftest_status = 0; + s.selftest_error = 0; + s.selftest_deadline = Some(s.clock_ns.saturating_add(1000)); + if fits && s.storage.get(&0x20038).copied().unwrap_or(0) == 0x80 { + self.memory.zero(selected as usize, size); + let start = selected * FITTED_BANK; + s.ecc + .retain(|key, _| !(*key >= start / 8 && *key < (start + size) / 8)); + let lines: Vec = s + .marked_directory + .iter() + .copied() + .filter(|&line| line >= start / 128 && line < (start + size) / 128) + .collect(); + for line in lines { + let word = *s.directory.get(&line).unwrap_or(&0); + s.directory + .insert(line, directory_encode(word, s.directory_config & 1 != 0)); + s.marked_directory.remove(&line); + } + } + Some(0) + } + fn scrub( + &self, + s: &mut State, + at: u64, + bytes: u64, + write: Option, + event: &mut Vec, + ) -> Option { + if write.is_none() { + return self.ecc(s, at, bytes, None); + } + if bytes != 8 || at >= 1 << 30 { + return self.refuse("scrub", at); + } + let software = at / (BANK_SPAN / 8); + let within = at % (BANK_SPAN / 8); + let address = software * BANK_SPAN + within * 8; + let Some((bank, offset)) = BedrockMemory::decode(s.config, address, 8) else { + return self.refuse("scrub", at); + }; + for i in 0..8 { + if BedrockMemory::decode(s.config, address + i * 8, 8).is_some() { + self.checked_word(s, bank, offset + i * 8, address + i * 8, event); + } + } + Some(0) + } + fn ecc(&self, s: &mut State, at: u64, bytes: u64, write: Option) -> Option { + if at >= 1 << 30 || (bytes != 1 && bytes != 2) { + return self.refuse("ecc", at); + } + let software = at / (BANK_SPAN / 8); + let within = (at % (BANK_SPAN / 8)) ^ if bytes == 1 { 7 } else { 6 }; + let address = software * BANK_SPAN + within * 8; + let Some((bank, offset)) = BedrockMemory::decode(s.config, address, bytes * 8) else { + if bytes == 2 && [0x04000006, 0x08000006].contains(&at) { + let i = usize::from(at == 0x08000006); + let old = s.admitted[i]; + if let Some(value) = write { + s.admitted[i] = value as u16; + } + return Some(old as u64); + } + return self.refuse("ecc", at); + }; + let mut old = 0; + for i in 0..bytes { + let raw = offset + i * 8; + let key = (bank as u64 * FITTED_BANK + raw) / 8; + let computed = ecc_compute(self.memory.raw_read(bank, raw)); + old |= (s.ecc.get(&key).copied().unwrap_or(computed) as u64) << (i * 8); + if let Some(value) = write { + let code = (value >> (i * 8)) as u8; + if code == computed { + s.ecc.remove(&key); + } else { + s.ecc.insert(key, code); + } + } + } + Some(old) + } +} +impl Resettable for BedrockMd { + fn power_on(&self) { + // Reset is called on the CPU thread or with it stopped. + let _remapping = self.remapping.lock(); + *self.state.lock() = State::default(); + self.memory.remap(0, BTreeSet::new(), false); + } +} +/// Resolve each DMA access through the atomic Physical table, as HEART +/// does. Do not ask Physical for mem_ptr or gen_ptr: those read the inline +/// CPU bitmap. Default block methods split accesses and respect slow pages +/// even if a caller supplies a block crossing a 4 KiB page boundary. +struct DmaView(Arc); +impl DmaView { + fn at(&self, address: u64, bytes: u64) -> Option<&dyn BusDevice> { + BedrockMemory::decode(self.0.memory.config(), address, bytes)?; + let phys = self.0.physical(); + let entry = phys.entry_at(address); + if crate::ppmem::devmap::entry_eq(entry, phys.default_entry()) { + return None; + } + // SAFETY: the owning MD retains Physical, which keeps every entry + // alive across remaps, including the weak MD slow endpoint. + Some(unsafe { &*entry }) + } +} +macro_rules! dma_rw { + ($read:ident, $write:ident, $result:ident, $ty:ty) => { + fn $read(&self, address: u64) -> $result { + self.at(address, std::mem::size_of::<$ty>() as u64) + .map_or($result::err(), |device| device.$read(address)) + } + fn $write(&self, address: u64, value: $ty) -> u32 { + self.at(address, std::mem::size_of::<$ty>() as u64) + .map_or(BUS_ERR, |device| { + let result = device.$write(address, value); + if result == BUS_OK { self.0.memory.written(address); } + result + }) + } + }; +} +impl BusDevice for DmaView { + dma_rw!(read8, write8, BusRead8, u8); + dma_rw!(read16, write16, BusRead16, u16); + dma_rw!(read32, write32, BusRead32, u32); + dma_rw!(read64, write64, BusRead64, u64); + #[cfg(feature = "jitv2")] + fn gen_ptr(&self, address: u64) -> *const AtomicU64 { + self.0.memory.gen_ptr(address) + } +} + +impl BusDevice for MdBus { + /// Bedrock's ECC view is the one device that sees the whole source register + /// of an uncached SH (`BusDevice::write_narrow_from`); every other view + /// takes the architectural low bits. + fn write_narrow_from(&self, addr: u64, size: usize, gpr: u64) -> u32 { + if size == 2 && matches!(self.view, View::Ecc) { + // PROM 6.210's memory test stores each check word with `sh` from a + // register whose low 16 bits are zero: the word sits elsewhere in + // the register, by the address of the memory it checks. Observed + // (memory address mod 0x40, carrier offset mod 8, register bits): + // 0x00 6 15:0 + // 0x10 4 63:48 + // 0x20 2 47:32 + // 0x30 0 31:16 + // The shift below gives exactly these four. Which lanes the R14000 + // drives for a narrow store, and how Bedrock wires them to the + // check bits, is unknown; this matches the PROM, not a hardware + // description. + let memory_address = (addr ^ 6).wrapping_mul(8); + return self.write16(addr, (gpr >> (memory_address.wrapping_neg() & 63)) as u16); + } + if size == 1 { self.write8(addr, gpr as u8) } else { self.write16(addr, gpr as u16) } + } + + #[cfg(feature = "jitv2")] + fn gen_ptr(&self, address: u64) -> *const std::sync::atomic::AtomicU64 { + let Some(md) = self.md.upgrade() else { + return std::ptr::null(); + }; + if !matches!(self.view, View::Memory | View::Uncached) { + return std::ptr::null(); + } + md.memory.gen_ptr(address) + } + + fn read8(&self, a: u64) -> BusRead8 { + self.md + .upgrade() + .and_then(|m| m.access(self.view, a, 1, None)) + .map_or(BusRead8::err(), |v| BusRead8::ok(v as u8)) + } + fn read16(&self, a: u64) -> BusRead16 { + self.md + .upgrade() + .and_then(|m| m.access(self.view, a, 2, None)) + .map_or(BusRead16::err(), |v| BusRead16::ok(v as u16)) + } + fn read32(&self, a: u64) -> BusRead32 { + self.md + .upgrade() + .and_then(|m| m.access(self.view, a, 4, None)) + .map_or(BusRead32::err(), |v| BusRead32::ok(v as u32)) + } + fn read64(&self, a: u64) -> BusRead64 { + self.md + .upgrade() + .and_then(|m| m.access(self.view, a, 8, None)) + .map_or(BusRead64::err(), BusRead64::ok) + } + fn write8(&self, a: u64, v: u8) -> u32 { + self.md + .upgrade() + .and_then(|m| m.access(self.view, a, 1, Some(v as u64))) + .map_or(BUS_ERR, |_| BUS_OK) + } + fn write16(&self, a: u64, v: u16) -> u32 { + self.md + .upgrade() + .and_then(|m| m.access(self.view, a, 2, Some(v as u64))) + .map_or(BUS_ERR, |_| BUS_OK) + } + fn write32(&self, a: u64, v: u32) -> u32 { + self.md + .upgrade() + .and_then(|m| m.access(self.view, a, 4, Some(v as u64))) + .map_or(BUS_ERR, |_| BUS_OK) + } + fn write64(&self, a: u64, v: u64) -> u32 { + self.md + .upgrade() + .and_then(|m| m.access(self.view, a, 8, Some(v))) + .map_or(BUS_ERR, |_| BUS_OK) + } +} +fn storage_mask(at: u64) -> Option { + Some(match at { + reg::ARBITRATION => 0xff, + reg::MIGRATION_CONFIG => 0x3ff | (0xff << 16) | (1 << 32), + reg::PERF_SELECT0 | reg::PERF_SELECT1 => { + 3 | (15 << 4) | (127 << 8) | (15 << 16) | (0xffffffffff << 20) | (1 << 63) + } + reg::PERF_COUNTER0 | reg::PERF_COUNTER1 => (1 << 41) - 1, + reg::JUNK_BUS_TIMING => 0xffffffffffff, + reg::SELFTEST_DATA => 0xff, + reg::ISSUE_DEBUG => 3, + reg::MIGRATION_DIFFERENCE => 0x7fff | (7 << 32) | (7 << 35) | (1 << 63), + reg::MIGRATION_VALUE => 0x7fff | (7 << 32) | (7 << 35) | (3 << 62), + reg::DIRECTORY_DEBUG => 3, + reg::MEMORY_DEBUG => 1, + _ => return None, + }) +} +fn idle(at: u64) -> bool { + matches!( + at, + 0x20 | 0x28 + | 0x40 + | 0x48 + | 0x58 + | 0x20010 + | 0x20018 + | 0x20020 + | 0x20028 + | 0x20030 + | 0x20040 + | 0x20048 + | 0x40020 + | 0x40028 + | 0x40030 + | 0x40038 + | 0x40058 + | 0x60020 + | 0x60028 + | 0x60030 + | 0x60038 + ) +} + +#[cfg(test)] +mod tests { + // Cases follow tests/qtest/sn1-bedrock-md-test.c in + // jamesbraid/qemu-sgi-origami: memory-banks, reset, queue-sizes, + // writable-masks, selftest-phases, fetchop-operations, ecc-carrier and + // directory-storage. + use super::*; + fn configured() -> Arc { + let md = BedrockMd::new(); + assert_eq!(md.md_bus().write64(8, 0x33), BUS_OK); + md + } + + /// Ordinary CPU and DMA RAM accesses remain available while MD state is locked. + #[test] + fn ordinary_cpu_and_dma_memory_do_not_take_the_md_lock() { + let md = configured(); + let dma = md.dma_view(); + let _state = md.state.lock(); + assert_eq!(md.physical().write64(0x1000, 0x1122334455667788), BUS_OK); + assert_eq!(md.physical().read64(0x1000).data, 0x1122334455667788); + assert_eq!(dma.read64(0x1000).data, 0x1122334455667788); + assert_eq!(dma.write32(0x1004, 0xabcdef01), BUS_OK); + assert_eq!(md.physical().read64(0x1000).data, 0x11223344abcdef01); + assert_eq!(dma.read64(FITTED_BANK).status, BUS_ERR); + } + + /// An ECC injection removes its direct mapping; DMA repair keeps the page slow. + #[test] + fn injected_checks_disable_the_window_and_dma_repair_keeps_the_slow_page() { + use crate::ppmem::MappedMemory; + let md = configured(); + let phys = md.physical(); + let space = phys.ppmem_space().expect("a window"); + assert_eq!(space.mapped_bitmap() & 3, 3); + assert!(phys.mem_ptr(0).is_some()); + assert_eq!(md.ecc_bus().write8(7, 0x13), BUS_OK); + assert_eq!(space.mapped_bitmap() & 3, 2, "the full 64 MiB region loses its direct bit"); + assert!(phys.mem_ptr(0).is_none()); + let slow = phys.entry_at(0); + assert!(!crate::ppmem::devmap::entry_eq(slow, phys.entry_at(0x1000))); + let dma = md.dma_view(); + let repaired = std::thread::spawn(move || dma.read64(0)).join().unwrap(); + assert_eq!(repaired.status, BUS_OK); + assert_eq!(repaired.data, 1); + assert!(md.state.lock().ecc.is_empty()); + assert_eq!(space.mapped_bitmap() & 3, 2, "a DMA thread never republishes the CPU bitmap"); + assert!(crate::ppmem::devmap::entry_eq(slow, phys.entry_at(0))); + // Adjacent pages still use the ordinary bank entry, even though + // the region no longer has a direct window mapping. + { + let _state = md.state.lock(); + assert_eq!(phys.write64(0x1000, 9), BUS_OK); + assert_eq!(phys.read64(0x1000).data, 9); + } + md.soft_reset(); + assert_eq!(space.mapped_bitmap() & 3, 3); + assert!(phys.mem_ptr(0).is_some()); + assert_eq!(phys.read64(0).data, 1); + } + + /// Directory-marked pages follow their physical bank after a configuration swap. + #[test] + fn marked_directory_pages_follow_their_bank_when_swizzled() { + use crate::ppmem::MappedMemory; + let md = configured(); + let phys = md.physical(); + let space = phys.ppmem_space().expect("a window"); + md.md_bus().write64(reg::DIRECTORY_CONFIG, 2); + md.directory_bus().write64(0, 0x001e0001); + assert_eq!(space.mapped_bitmap() & 1, 0); + assert!(phys.mem_ptr(0).is_none()); + md.md_bus().write64(reg::CONFIG, 0x33 | (1 << 32)); + assert_ne!(space.mapped_bitmap() & 1, 0, "the other bank now answers at zero"); + assert_eq!(space.mapped_bitmap() & (1 << 16), 0); + assert!(phys.mem_ptr(BANK_SPAN).is_none()); + md.md_bus().write64(reg::DIRECTORY_CONFIG, 0); + assert_eq!(phys.read64(BANK_SPAN).status, BUS_OK); + assert_ne!(md.md_bus().read64(reg::DIRECTORY_ERROR).data & (1 << 61), 0); + md.rebuild_memory(); + assert!(phys.mem_ptr(BANK_SPAN).is_some()); + md.power_on(); + assert_eq!(space.mapped_bitmap(), 0); + assert_eq!(phys.read64(BANK_SPAN).status, BUS_ERR); + } + + /// Guest code must not keep a compiled page after its backing bank changes. + /// page_for deliberately retains its generation pointer. A remap must + /// invalidate that page and subsequent writes to its new bank must keep + /// invalidating the same pointer, including after swapping back. + #[cfg(feature = "jitv2")] + #[test] + fn a_claimed_jit_page_tracks_physical_ownership_across_bank_swaps() { + use crate::cpu::jitv2::jitv2::Jitv2; + let md = configured(); + let dma = md.dma_view(); + let mut jit = Jitv2::new(4); + let slot = jit.page_for(0, 0, dma.as_ref(), false).unwrap(); + let page = jit.page_ptr(slot); + let generation = || unsafe { (*page).current_gen() }; + let pointer = dma.gen_ptr(0); + let mut entries = [0u64; crate::cpu::jitv2::jitv2::BITMAP_WORDS]; + entries[0] = 1; + // Exercise the JIT publication/invalidation protocol without + // executing a synthetic function body. + let code = std::ptr::NonNull::<()>::dangling().as_ptr() as *const (); + assert!(unsafe { (*page).publish(&entries, code, generation(), 1, 0, false) }); + assert!(unsafe { (*page).is_runnable(0) }); + let first = generation(); + md.md_bus().write64(reg::CONFIG, 0x33 | (1 << 32)); + assert_eq!(jit.page_for(0, 0, dma.as_ref(), false), Some(slot)); + assert_eq!(dma.gen_ptr(0), pointer); + assert!(generation() > first); + assert!(!unsafe { (*page).is_runnable(0) }); + let swapped = generation(); + assert!(unsafe { (*page).publish(&entries, code, swapped, 1, 0, false) }); + assert!(unsafe { (*page).is_runnable(0) }); + assert_eq!(dma.write64(0, 0x11223344), BUS_OK); + assert!(generation() > swapped); + assert!(!unsafe { (*page).is_runnable(0) }, "code published after a swap sees new-bank writes"); + assert_eq!(md.physical().banks()[1].read64(0).data, 0x11223344); + let written = generation(); + md.md_bus().write64(reg::CONFIG, 0x33); + assert!(generation() > written, "swapping back cannot lower the logical generation"); + let restored = generation(); + md.uncached_view().write64(0, 0x55667788); + assert!(generation() > restored); + let before_fetchop = generation(); + md.fetchop_bus().write64(0, 0); + assert!(generation() > before_fetchop); + md.ecc_bus().write8(7, 0x13); + let before_repair = generation(); + assert_eq!(dma.read64(0).data, 1); + assert!(generation() > before_repair, "ECC repair invalidates logical code pages"); + md.md_bus().write64(reg::CONFIG, 0x13); + md.md_bus().write64(0x20038, 0x80); + let before_zero = generation(); + md.md_bus().write64(reg::SELFTEST_COMMAND, 1); + assert!(generation() > before_zero, "zero self-test invalidates logical code pages"); + let before_disable = generation(); + md.power_on(); + assert!(generation() > before_disable); + assert_eq!(dma.gen_ptr(0), pointer, "a disabled bank keeps its logical counter alive"); + } + + /// Guest density settings remap fitted RAM, and reset restores the default map. + #[test] + fn bank_configuration_remaps_memory_and_reset_restores_defaults() { + let md = configured(); + let bus = md.physical(); + assert_eq!(bus.write64(0, 0x1122334455667788), BUS_OK); + assert_eq!(bus.write64(BANK_SPAN, 0xabcdef), BUS_OK); + assert_eq!(bus.read8(0).data, 0x11); + assert_eq!(bus.read32(4).data, 0x55667788); + assert_eq!(bus.read64(FITTED_BANK).status, BUS_ERR); + assert_eq!(bus.read64(2 * BANK_SPAN).status, BUS_ERR); + assert_eq!(md.md_bus().write64(8, 0x33 | 1 << 32), BUS_OK); + assert_eq!(bus.read64(0).data, 0xabcdef); + assert_eq!(bus.read64(BANK_SPAN).data, 0x1122334455667788); + assert!(bus.mem_ptr(0).is_some()); + md.soft_reset(); + assert_eq!(bus.read64(0).data, 0xabcdef); + md.power_on(); + assert_eq!(bus.read64(0).status, BUS_ERR); + bus.map_unmapped(0x1fc00000, 0x1fc01000); + for (code, size) in [(1, 64 << 20), (2, 128 << 20), (3, FITTED_BANK), + (4, FITTED_BANK), (5, FITTED_BANK)] { + md.md_bus().write64(reg::CONFIG, 1 | (code << 4)); + assert_eq!(bus.read64(size - 8).status, BUS_OK); + assert_eq!(bus.read64(size).status, BUS_ERR); + assert_eq!(bus.read64(0x1fc00000).status, BUS_OK, "a non-RAM mapping survives"); + } + md.md_bus().write64(reg::CONFIG, 0x11); + assert_eq!(bus.read64(64 << 20).status, BUS_ERR, "the old prefix is removed on shrink"); + md.md_bus().write64(reg::CONFIG, 0x61); + assert_eq!(bus.read64(0).status, BUS_ERR, "unsupported density does not map RAM"); + #[cfg(feature = "jitv2")] + { + let md = configured(); + let ram = md.physical(); + let first = ram.gen_ptr(0); + let second = ram.gen_ptr(BANK_SPAN); + assert!(!first.is_null() && !second.is_null()); + assert_ne!(first, second); + let before = unsafe { (*first).load(Ordering::Relaxed) }; + ram.write64(0, 0x55); + assert!(unsafe { (*first).load(Ordering::Relaxed) } > before); + md.md_bus().write64(8, 0x33 | (1 << 32)); + // The window address stays fixed while its backing bank changes. + assert_eq!(ram.gen_ptr(0), first); + assert_eq!(ram.gen_ptr(BANK_SPAN), second); + let bank0 = ram.banks()[0].gen_ptr(0); + let bank1 = ram.banks()[1].gen_ptr(0); + let before0 = unsafe { (*bank0).load(Ordering::Relaxed) }; + let before1 = unsafe { (*bank1).load(Ordering::Relaxed) }; + assert_eq!(ram.write64(0, 0xaa), BUS_OK); + assert_eq!(unsafe { (*bank0).load(Ordering::Relaxed) }, before0); + assert_eq!(unsafe { (*bank1).load(Ordering::Relaxed) }, before1 + 1); + assert_eq!(unsafe { (*first).load(Ordering::Relaxed) }, before1 + 1); + let view = md.uncached_view(); + let logical = view.gen_ptr(0); + assert_eq!(logical, md.memory.gen_ptr(0)); + let logical_before = unsafe { (*logical).load(Ordering::Relaxed) }; + drop(md); + assert_eq!(unsafe { (*logical).load(Ordering::Relaxed) }, logical_before, + "an external endpoint keeps its logical generation pointers alive"); + // The endpoint keeps the backing memory alive after MD disappears. + assert!(unsafe { (*bank1).load(Ordering::Relaxed) } > 0); + assert_eq!(view.read64(0).status, BUS_ERR); + } + + } + /// The guest reads ECC configuration and memory or directory errors through status registers. + #[test] + fn guest_registers_expose_ecc_configuration_and_error_latches() { + { + let md = configured(); + let r = md.md_bus(); + // These carrier offsets are idle; they do not select storage. + for at in [0x20, 0x20010, 0x40020, 0x60020] { + let before = md.unsupported(); + assert_eq!(r.write64(at, u64::MAX), BUS_OK, "{at:#x}"); + assert_eq!(r.read64(at).data, 0, "{at:#x}"); + assert_eq!(md.unsupported(), before + 1); + } + for (at, value) in [(0x60, 0x001818180c0c0c00), (0x40050, 0x04040400)] { + let before = md.unsupported(); + assert_eq!(r.write64(at, 0), BUS_OK); + assert_eq!(r.read64(at).data, value); + assert_eq!(md.unsupported(), before + 1); + } + // Error-clear writes are ignored too: only a read clears the record. + md.ecc_bus().write8(7, 3); + md.uncached_view().read64(0); + r.write64(0x40000, 2); + md.directory_bus().write64(0, 3 << 16); + r.write64(0x40000, 0); + md.physical().read64(0); + let memory_error = r.read64(0x60010).data; + let directory_error = r.read64(0x40010).data; + assert_ne!(memory_error & (1 << 60), 0); + assert_ne!(directory_error & (1 << 63), 0); + for at in [0x40010, 0x40018, 0x60010, 0x60018] { + let before = md.unsupported(); + assert_eq!(r.write64(at, u64::MAX), BUS_OK); + assert_eq!(r.read64(0x60010).data, memory_error); + assert_eq!(r.read64(0x40010).data, directory_error); + assert_eq!(md.unsupported(), before + 1); + } + // An unsupported self-test command leaves an in-flight command alone. + r.write64(0x20030, 1); + let before = md.unsupported(); + assert_eq!(r.write64(0x20030, 3), BUS_OK); + assert_eq!(md.unsupported(), before + 1); + md.advance_ns(1000); + assert_eq!(r.read64(0x20048).data, 3); + assert_eq!(r.write64(0x70000, 1), BUS_ERR); + assert_eq!(r.read64(0x70000).status, BUS_ERR); + assert_eq!(r.write32(0x60, 0), BUS_ERR); + assert_eq!(r.write64(0x61, 0), BUS_ERR); + } + { + let md = configured(); + let regs = md.md_bus(); + assert_eq!(regs.read64(0x60000).data, 0); + assert_eq!(regs.write64(0x60000, u64::MAX), BUS_OK); + assert_eq!(regs.read64(0x60000).data, 1); + assert_eq!(regs.write64(0x60000, !1), BUS_OK); + assert_eq!(regs.read64(0x60000).data, 0); + assert_eq!(regs.write8(0x60000, 1), BUS_ERR); + assert_eq!(regs.write64(0x60001, 1), BUS_ERR); + regs.write64(0x60000, 1); + md.soft_reset(); + assert_eq!(regs.read64(0x60000).data, 0); + assert_eq!(regs.read64(8).data, 0x33); + regs.write64(0x60000, 1); + md.power_on(); + assert_eq!(regs.read64(0x60000).data, 0); + assert_eq!(regs.read64(8).data, 0); + } + { + for (code, kind, status, valid) in [ + (1, MemoryError::Correctable, BUS_OK, 1 << 52), + (3, MemoryError::Uncorrectable, BUS_ERR, 1 << 60), + ] { + let md = configured(); + let regs = md.md_bus(); + let ecc = md.ecc_bus(); + let ram = md.uncached_view(); + let events = Arc::new(Mutex::new(Vec::new())); + let seen = events.clone(); + md.on_error(Arc::new(move |event| seen.lock().push(event))); + assert_eq!(ram.read64(0).status, BUS_OK); + ecc.write8(7, code); + regs.write64(0x60000, u64::MAX); + assert_eq!(ram.read64(0).status, BUS_OK); + assert_eq!(ecc.read8(7).data, code); + assert_eq!(regs.read64(0x60010).data, 0); + assert!(events.lock().is_empty()); + regs.write64(0x60000, !1); + assert_eq!(regs.read64(0x60000).data, 0); + assert_eq!(ram.read64(0).status, status); + assert_ne!(regs.read64(0x60010).data & valid, 0); + assert_eq!(*events.lock(), vec![kind]); + if kind == MemoryError::Correctable { + assert_eq!(ecc.read8(7).data, 0); + assert_eq!(ram.read64(0).status, BUS_OK); + assert_eq!(*events.lock(), vec![kind]); + } + } + } + { + let md = configured(); + let regs = md.md_bus(); + let ecc = md.ecc_bus(); + let ram = md.uncached_view(); + ecc.write8(7, 3); + regs.write64(0x60000, 1); + assert_eq!(ram.write8(7, 0x5a), BUS_OK); + assert_eq!(ecc.read8(7).data, 0x82); + regs.write64(0x60000, 0); + assert_eq!(ram.read64(0).status, BUS_OK); + assert_eq!(ram.read64(0).data, 0x5a); + assert_eq!(regs.read64(0x60010).data, 0); + } + } + /// Self-test completion is visible only after its configured clock interval. + #[test] + fn selftest_waits_for_fitted_clock_cycles_and_zero_fills_ram() { + { + let now = Arc::new(AtomicU64::new(100)); + let time = now.clone(); + let md = BedrockMd::with_clock(Arc::new(move || time.load(Ordering::Relaxed))); + let regs = md.md_bus(); + regs.write64(8, 0x33); + md.start_clock(); + regs.write64(0x20030, 1); + now.store(1099, Ordering::Relaxed); + assert_eq!(regs.read64(0x20048).data, 0); + now.store(1100, Ordering::Relaxed); + assert_eq!(regs.read64(0x20048).data, 3); + now.store(5000, Ordering::Relaxed); + regs.write64(0x20030, 1); + now.store(5500, Ordering::Relaxed); + md.stop_clock(); + now.store(15000, Ordering::Relaxed); + assert_eq!(regs.read64(0x20048).data, 0); + md.start_clock(); + now.store(15499, Ordering::Relaxed); + assert_eq!(regs.read64(0x20048).data, 0); + now.store(15500, Ordering::Relaxed); + assert_eq!(regs.read64(0x20048).data, 3); + regs.write64(0x20030, 1); + now.store(15400, Ordering::Relaxed); + assert_eq!(regs.read64(0x20048).data, 0); + now.store(16499, Ordering::Relaxed); + assert_eq!(regs.read64(0x20048).data, 0); + now.store(16500, Ordering::Relaxed); + assert_eq!(regs.read64(0x20048).data, 3); + regs.write64(0x20030, 1); + regs.write64(0x20030, 2); + now.store(20000, Ordering::Relaxed); + assert_eq!(regs.read64(0x20048).data, 0); + md.power_on(); + regs.write64(8, 0x33); + md.start_clock(); + regs.write64(0x20030, 1); + now.store(20999, Ordering::Relaxed); + assert_eq!(regs.read64(0x20048).data, 0); + now.store(21000, Ordering::Relaxed); + assert_eq!(regs.read64(0x20048).data, 3); + } + { + let md = configured(); + let r = md.md_bus(); + r.write64(0x20030, 0x21); + md.advance_ns(1000); + assert_eq!(r.read64(0x20048).data, 2); + assert_eq!(r.read64(0x20040).data, 1 << 48); + r.write64(0x20030, 1); + r.write64(0x20030, 2); + md.advance_ns(1000); + assert_eq!(r.read64(0x20048).data, 0); + assert_eq!(r.write64(0x20030, 3), BUS_OK); + } + { + let md = configured(); + let r = md.md_bus(); + r.write64(8, 0x13); + md.physical().write64(0, 0x1234); + md.ecc_bus().write8(7, 1); + r.write64(0x20038, 0x80); + r.write64(0x20030, 1); + md.advance_ns(1000); + assert_eq!(r.read64(0x20048).data, 3); + assert_eq!(md.physical().read64(0).data, 0); + assert_eq!(md.ecc_bus().read8(7).data, 0); + assert_eq!(r.read64(0x60010).data, 0); + } + } + /// Fetch-and-op registers preserve the big-endian lanes used by the processor. + #[test] + fn fetchop_preserves_big_endian_word_lanes() { + let md = configured(); + let ram = md.physical(); + let f = md.fetchop_bus(); + ram.write64(0, 0xffffffff00000009); + assert_eq!(f.read32(8).data, u32::MAX); + assert_eq!(ram.read64(0).data, 9); + assert_eq!(f.read32(12).data, 9); + assert_eq!(ram.read64(0).data, 10); + assert_eq!(f.write32(28, 3), BUS_OK); + assert_eq!(ram.read64(0).data, 2); + assert_eq!(f.write64(32, 8), BUS_OK); + assert_eq!(ram.read64(0).data, 10); + assert_eq!(f.read64(24).data, 10); + assert_eq!(ram.read64(0).data, 0); + assert_eq!(f.read64(32).status, BUS_ERR); + assert_eq!(f.write64(40, 1), BUS_ERR); + assert_eq!(f.read16(0).status, BUS_ERR); + } + /// The guest reads ECC bytes and scrubbing repairs correctable memory. + #[test] + fn ecc_carriers_expose_check_bytes_and_scrub_corrections() { + { + let md = configured(); + let ecc = md.ecc_bus(); + for carrier in 0..8 { + let source = 0x11223344556677a5u64; + assert_eq!(ecc.write8(carrier, source as u8), BUS_OK); + assert_eq!(ecc.read8(carrier).data, source as u8, "carrier {carrier}"); + } + assert_eq!(ecc.write16(4, 0x1234), BUS_OK); + assert_eq!(ecc.read16(4).data, 0x1234); + assert_eq!(ecc.write16(6, 0x2313), BUS_OK); + assert_eq!(ecc.read16(6).data, 0x2313); + assert_eq!(ecc.write16(0x04000006, 0xface), BUS_OK); + assert_eq!(ecc.read16(0x04000006).data, 0xface); + + } + { + let md = configured(); + let events = Arc::new(Mutex::new(Vec::new())); + let seen = events.clone(); + let weak = Arc::downgrade(&md); + md.on_error(Arc::new(move |kind| { + assert_eq!(weak.upgrade().unwrap().current_cell(), u64::MAX); + seen.lock().push(kind); + })); + assert_eq!(md.ecc_bus().write8(7, 3), BUS_OK); + assert_eq!(md.ecc_bus().write8(6, 0x13), BUS_OK); + assert_eq!(md.ecc_bus().write8(5, 0x23), BUS_OK); + assert_eq!(md.ecc_scrub_bus().write64(0, 0), BUS_OK); + assert_eq!( + *events.lock(), + vec![ + MemoryError::Uncorrectable, + MemoryError::Correctable, + MemoryError::Correctable, + ] + ); + let record = md.md_bus().read64(0x60010).data; + assert_ne!(record & (1 << 60), 0); + assert_ne!(record & (1 << 52), 0); + assert_ne!(record & (1 << 53), 0); + assert_eq!((record >> 40) & 0xff, 3); + assert_eq!(record & (((1u64 << 30) - 1) << 3), 0); + assert_eq!(md.ecc_bus().read8(6).data, 0x13); + assert_eq!(md.ecc_bus().read8(5).data, 0x23); + } + } + /// The ECC view takes an uncached SH's check word from wherever the PROM + /// put it in the source register; other views, and bytes, take the low bits. + #[test] + fn ecc_view_reads_the_check_word_from_the_whole_source_register() { + let md = configured(); + let ecc = md.ecc_bus(); + // (carrier offset, register bits holding the check word) + for (carrier, shift) in [(6u64, 0u32), (4, 48), (2, 32), (0, 16)] { + let source = 0x1111_2222_3333_4444u64 ^ (0xa55a_u64 << shift); + let word = (source >> shift) as u16; + assert_eq!(ecc.write_narrow_from(carrier, 2, source), BUS_OK); + assert_eq!(ecc.read16(carrier).data, word, "carrier {carrier}"); + } + assert_eq!(ecc.write_narrow_from(5, 1, 0xfeed_0000_0000_0023), BUS_OK); + assert_eq!(ecc.read8(5).data, 0x23, "a byte keeps the architectural low bits"); + // The memory view is not the ECC view: its halfword is the low bits. + let view = md.uncached_view(); + assert_eq!(view.write_narrow_from(0x40, 2, 0xdead_beef_1234_5678), BUS_OK); + assert_eq!(md.dma_view().read64(0x40).data >> 48, 0x5678); + } + /// Uncached CPU and DMA views report uncorrectable errors with bus status. + #[test] + fn uncached_and_dma_views_report_errors_to_the_guest() { + let md = configured(); + let r = md.md_bus(); + let e = md.ecc_bus(); + e.write8(7, 3); + assert_eq!(md.uncached_view().read64(0).status, BUS_ERR); + assert_eq!(md.dma_view().read64(0).status, BUS_OK); + assert_ne!(r.read64(0x60018).data & (1 << 60), 0); + e.write8(7, 0x13); + assert_eq!(md.ecc_scrub_bus().write64(0, 0), BUS_OK); + assert_eq!(md.physical().read64(0).data, 1); + assert_eq!(e.read8(7).data, 0x13); + assert_eq!(md.ecc_scrub_bus().write32(0, 0), BUS_ERR); + } + /// Directory reads report invalid metadata and protection faults to the guest. + #[test] + fn directory_reads_check_metadata_and_report_protection_errors() { + { + let md = configured(); + let events = Arc::new(Mutex::new(Vec::new())); + let seen = events.clone(); + let weak = Arc::downgrade(&md); + md.on_error(Arc::new(move |kind| { + assert_eq!(weak.upgrade().unwrap().current_cell(), u64::MAX); + seen.lock().push(kind); + })); + let r = md.md_bus(); + md.directory_bus().write64(0, 1); + assert_eq!(md.directory_bus().read64(0).data, 0x001f0001); + r.write64(0x40000, 2); + md.directory_bus().write64(0, 0x001e0001); + r.write64(0x40000, 0); + md.ecc_bus().write8(7, 3); + assert_eq!(md.uncached_view().read64(0).status, BUS_ERR); + assert_eq!(md.directory_bus().read64(0).data, 0x001f0001); + assert_eq!( + *events.lock(), + vec![MemoryError::Correctable, MemoryError::Uncorrectable] + ); + assert_ne!(r.read64(0x40010).data & (1 << 61), 0); + assert_ne!(r.read64(0x60010).data & (1 << 60), 0); + } + { + let md = configured(); + let d = md.directory_bus(); + assert_eq!(d.write64(0x408, 0x1234), BUS_OK); + assert_eq!(d.read64(0x2000408).data, 0x1234); + assert_eq!(d.write64(0x418, 0x5678), BUS_OK); + assert_eq!(d.read64(0x408).data, 0x1234); + assert_eq!(d.read64(0x418).data, 0x5678); + assert_eq!(d.write64(8, u64::MAX), BUS_OK); + assert_eq!(d.read64(8).data, u64::MAX); + assert_eq!(d.read32(0).status, BUS_ERR); + assert_eq!(d.read64(1 << 30).status, BUS_ERR); + } + } +} + +const DIRECTORY_PREMIUM: [u8; 64] = [ + 0x1f, 0x2f, 0x37, 0x3b, 0x3d, 0x3e, 0x07, 0x0b, 0x0d, 0x0e, 0x13, 0x15, 0x16, 0x19, 0x1a, 0x1c, + 0x7f, 0x4f, 0x57, 0x5b, 0x5d, 0x5e, 0x67, 0x6b, 0x6d, 0x6e, 0x73, 0x75, 0x76, 0x79, 0x7a, 0x7c, + 1, 2, 4, 8, 16, 32, 0x23, 0x25, 0x26, 0x29, 0x2a, 0x2c, 0x31, 0x32, 0x34, 0x38, 0x40, 0x43, + 0x45, 0x46, 0x49, 0x4a, 0x4c, 0x51, 0x52, 0x54, 0x58, 0x61, 0x62, 0x64, 0x68, 0x70, +]; +const DIRECTORY_STANDARD: [u8; 32] = [ + 0x1f, 0x2f, 0x37, 0x3b, 0x3d, 0x3e, 0x07, 0x0b, 0x0d, 0x0e, 0x13, 0x15, 0x16, 0x19, 0x1a, 0x1c, + 1, 2, 4, 8, 16, 32, 0x23, 0x25, 0x26, 0x29, 0x2a, 0x2c, 0x31, 0x32, 0x34, 0x38, +]; +fn directory_columns(premium: bool) -> &'static [u8] { + if premium { + &DIRECTORY_PREMIUM + } else { + &DIRECTORY_STANDARD + } +} +fn directory_syndrome(mut word: u64, premium: bool) -> u8 { + if !premium { + word &= u32::MAX as u64; + } + let mut check = 0; + while word != 0 { + check ^= directory_columns(premium)[word.trailing_zeros() as usize]; + word &= word - 1; + } + check +} +pub fn directory_encode(word: u64, premium: bool) -> u64 { + let word = word + & !if premium { + (63 << 32) | (1 << 48) + } else { + 63 << 16 + }; + let check = directory_syndrome(word, premium) as u64; + word | if premium { + (check & 63) << 32 | (check >> 6) << 48 + } else { + check << 16 + } +} diff --git a/src/dev/bedrock_mem.rs b/src/dev/bedrock_mem.rs new file mode 100644 index 000000000..13494a028 --- /dev/null +++ b/src/dev/bedrock_mem.rs @@ -0,0 +1,232 @@ +//! The Fuel's fitted memory: two 256 MiB banks owned by `Physical`, with +//! the bank decode selected by Bedrock's memory configuration register. +//! +//! This population follows the QEMU model; it is not a measured Fuel +//! topology. Ordinary pages answer through Physical's bank views, without +//! the MD state lock. Pages with injected ECC or marked directory lines use +//! the MD endpoint. Their entire 64 MiB region is absent from the direct +//! window, so cache fills cannot bypass those checks. +//! +//! Remapping runs on the CPU thread, or with the CPU stopped, as HEART's +//! memory descriptors do. DMA may remove diagnostic metadata but retains +//! the slow mapping until the next CPU-side rebuild. Physical publishes +//! complete decode-table pages atomically; a concurrent bus master sees an +//! old or new entry. This does not promise a transaction across a remap. +//! +//! Ported from hw/misc/sgi-bedrock-md.c in jamesbraid/qemu-sgi-origami. + +use crate::physical::{Physical, RamBank}; +use crate::traits::*; +use crate::ppmem::devmap::SLOT_SHIFT; +use crate::ppmem::{MappedMemory, BITMAP_SHIFT}; +use parking_lot::Mutex; +use std::collections::BTreeSet; +use std::sync::atomic::{AtomicBool, AtomicU64, Ordering}; +use std::sync::Arc; + +pub const BANK_SPAN: u64 = 1 << 30; +pub const FITTED_BANK: u64 = 256 << 20; + +#[derive(Clone, Copy, PartialEq, Eq)] +struct Run { + bank: usize, + base: u64, + size: u64, +} + +#[derive(Default)] +struct Mapping { + runs: Vec, + // Bank-relative 4 KiB page numbers, bank 1 immediately after bank 0. + slow_pages: BTreeSet, + slow: Option>, +} + +struct NoMemory; +impl BusDevice for NoMemory {} + +/// JIT pages belong to physical addresses, not to the bank currently +/// answering there. Keep these counters separate from the banks' counters: +/// a compiled page pins its pointer across later bank reconfiguration. +#[cfg(feature = "jitv2")] +pub(crate) struct Generations(Box<[AtomicU64]>); +#[cfg(feature = "jitv2")] +impl Generations { + fn new() -> Self { + Self((0..1 << 20).map(|_| AtomicU64::new(0)).collect::>().into_boxed_slice()) + } + pub(crate) fn base(&self) -> *mut AtomicU64 { + self.0.as_ptr() as *mut AtomicU64 + } + fn bump(&self, address: u64) { + if let Some(counter) = self.0.get((address >> 12) as usize) { + counter.fetch_add(1, Ordering::Relaxed); + } + } + fn bump_all(&self) { + for slot in 0..4 { + let first = slot * (BANK_SPAN >> 12) as usize; + for counter in &self.0[first..first + (FITTED_BANK >> 12) as usize] { + counter.fetch_add(1, Ordering::Relaxed); + } + } + } +} + +pub struct BedrockMemory { + physical: Arc, + config: AtomicU64, + mapping: Mutex, + #[cfg(feature = "jitv2")] + generations: Arc, +} +impl BedrockMemory { + pub fn new() -> Self { + let physical = Physical::new(vec![RamBank::new(256), RamBank::new(256)]); + physical.set_default(Arc::new(NoMemory), "Bedrock unconfigured memory"); + Self { physical, config: AtomicU64::new(0), mapping: Mutex::new(Mapping::default()), + #[cfg(feature = "jitv2")] + generations: Arc::new(Generations::new()), + } + } + pub fn physical(&self) -> &Arc { + &self.physical + } + pub(crate) fn config(&self) -> u64 { + self.config.load(Ordering::Acquire) + } + + #[cfg(feature = "jitv2")] + pub(crate) fn generations(&self) -> Arc { self.generations.clone() } + #[cfg(feature = "jitv2")] + pub(crate) fn gen_ptr(&self, address: u64) -> *const AtomicU64 { + // Disabled RAM must still have a counter: a JIT page may already + // have claimed it before the guest enables or moves a bank. + if address / BANK_SPAN < 4 && address % BANK_SPAN < FITTED_BANK { + unsafe { self.generations.base().add((address >> 12) as usize) } + } else { std::ptr::null() } + } + pub(crate) fn written(&self, _address: u64) { + #[cfg(feature = "jitv2")] + self.generations.bump(_address); + } + + /// The endpoint is weak back to MD: Physical owns it without a cycle. + pub(crate) fn set_slow_view(&self, slow: Arc) { + self.mapping.lock().slow = Some(slow); + } + + fn runs(config: u64) -> Vec { + (0..8).filter_map(|slot| { + let base = slot * BANK_SPAN; + let (bank, _) = Self::decode(config, base, 1)?; + let code = (config >> (bank / 2 * 8 + 4)) & 15; + Some(Run { bank, base, size: FITTED_BANK.min((32 << 20) << code) }) + }).collect() + } + + /// Publish a state snapshot after releasing MD's lock. Only the CPU + /// thread, or a caller with the CPU stopped, may change ppmem's bitmap. + /// A forced rebuild follows the machine setting Physical's default. + pub(crate) fn remap(&self, config: u64, slow_pages: BTreeSet, force: bool) { + let runs = Self::runs(config); + let mut old = self.mapping.lock(); + if !force && old.runs == runs && old.slow_pages == slow_pages { + self.config.store(config, Ordering::Release); + return; + } + let phys = &self.physical; + let space = phys.ppmem_space(); + // Withdraw the direct window before publishing any checked page. + if let Some(space) = space { + space.clear_mappings(); + } + // Restore only ranges previously owned by RAM. In particular, the + // unused part of a 1 GiB slot may contain flash or other devices. + for run in &old.runs { + let retained = runs.iter().find(|next| next.base == run.base) + .map_or(0, |next| next.size.min(run.size)); + if retained < run.size { + phys.fill_slots((run.base + retained) >> SLOT_SHIFT, + (run.size - retained) >> SLOT_SHIFT, phys.default_entry()); + } + } + for run in &runs { + phys.fill_slots(run.base >> SLOT_SHIFT, run.size >> SLOT_SHIFT, phys.bank_view(run.bank, run.base)); + let bank_page = run.bank as u64 * (FITTED_BANK >> SLOT_SHIFT); + let pages = run.size >> SLOT_SHIFT; + let mut checked_regions = BTreeSet::new(); + for &page in slow_pages.range(bank_page..bank_page + pages) { + let offset = (page - bank_page) << SLOT_SHIFT; + let address = run.base + offset; + checked_regions.insert(offset >> BITMAP_SHIFT); + phys.map(address, address + (1 << SLOT_SHIFT), + old.slow.as_ref().expect("Bedrock MD slow endpoint").clone(), "Bedrock checked RAM"); + } + if let Some(space) = space { + let region_size = 1 << BITMAP_SHIFT; + for offset in (0..run.size).step_by(region_size as usize) { + if !checked_regions.contains(&(offset >> BITMAP_SHIFT)) { + if let Err(error) = space.map_range(run.bank, offset, run.base + offset, region_size) { + crate::dlog!(crate::devlog::LogModule::Bedrock, + "Bedrock RAM window at {:#x}: {error}", run.base + offset); + } + } + } + } + } + self.config.store(config, Ordering::Release); + // Publish changed ownership before advancing the stable counters, + // so an in-flight compile cannot accept bytes from the old decode. + self.changed_decode(); + old.runs = runs; + old.slow_pages = slow_pages; + } + + pub(crate) fn decode(config: u64, addr: u64, bytes: u64) -> Option<(usize, u64)> { + let software = addr / BANK_SPAN; + if software >= 8 { + return None; + } + let bank = software ^ ((config >> 32) & 3); + let field = config >> (bank / 2 * 8); + let code = (field >> 4) & 15; + if bank >= 2 || field & (1 << (bank & 1)) == 0 || !(1..=5).contains(&code) { + return None; + } + let limit = FITTED_BANK.min((32 << 20) << code); + let offset = addr % BANK_SPAN; + (offset < limit && bytes <= limit - offset).then_some((bank as usize, offset)) + } + pub(crate) fn raw_read(&self, bank: usize, at: u64) -> u64 { + self.physical.banks()[bank].read64(at).data + } + pub(crate) fn raw_write(&self, bank: usize, at: u64, value: u64) -> u32 { + let result = self.physical.banks()[bank].write64(at, value); + if result == BUS_OK { + let slot = bank as u64 ^ ((self.config() >> 32) & 3); + self.written(slot * BANK_SPAN + at); + } + result + } + pub(crate) fn zero(&self, bank: usize, size: u64) { + assert!(size <= FITTED_BANK); + // The bank owns this contiguous mapping; zero has the same representation + // in host word order. Self-test does not synchronize concurrent bank accesses. + unsafe { + std::ptr::write_bytes(self.physical.banks()[bank].host_ptr(), 0, size as usize); + } + #[cfg(feature = "jitv2")] + self.physical.banks()[bank].bump_gen_all(); + #[cfg(feature = "jitv2")] + self.generations.bump_all(); + } + pub(crate) fn changed_decode(&self) { + #[cfg(feature = "jitv2")] + self.generations.bump_all(); + #[cfg(feature = "jitv2")] + for bank in self.physical.banks() { + bank.bump_gen_all(); + } + } +} diff --git a/src/dev/bedrock_ni.rs b/src/dev/bedrock_ni.rs new file mode 100644 index 000000000..06f1ee57c --- /dev/null +++ b/src/dev/bedrock_ni.rs @@ -0,0 +1,335 @@ +//! Bedrock NI exposes node routing, sideband and error registers. Fuel runs as +//! one node without a NUMAlink peer, but the guest still reads and programs +//! these local control words during machine setup. +//! +//! # What is approximated here +//! +//! Routing and sideband writes are retained; the network fabric and packet +//! delivery are not simulated. Unspecified power-on fields start at zero, and +//! the sideband word survives reset as the current compatibility behavior. +//! +//! Ported from hw/misc/sgi-bedrock-ni.c and hw/misc/sgi-bedrock-ni-regs.c +//! in jamesbraid/qemu-sgi-origami. + +/// Length of the block-relative register aperture. +pub const SIZE: u64 = 0x2008; + +pub mod reg { + pub const NODE_ID: u64 = 0x0; + pub const LOCAL_RESET: u64 = 0x8; + pub const CONTROL_0: u64 = 0x10; + pub const CONTROL_1: u64 = 0x18; + pub const CONTROL_2: u64 = 0x20; + pub const CONTROL_3: u64 = 0x30; + pub const CONTROL_4: u64 = 0x38; + pub const ERROR_STATUS: u64 = 0x100; + pub const ERROR_STATUS_UPPER: u64 = 0x108; + pub const ERROR_STATUS_LOWER: u64 = 0x110; + pub const SIDE_BAND: u64 = 0x118; + pub const ERROR_CLEAR: u64 = 0x120; + pub const ROUTES: u64 = 0x1000; + pub const ROUTES_END: u64 = 0x1400; + pub const GLOBAL_CONTROL: u64 = 0x2000; +} + +use crate::traits::{BusDevice, BusRead64, Resettable, BUS_ERR, BUS_OK}; +use parking_lot::{Mutex, ReentrantMutex}; +use std::sync::Arc; +type Level = Arc; +#[derive(Clone)] +struct Regs { + control: [u64; 8], + routes: [u64; 128], + global: u64, + errors: u64, + sideband: u64, +} +impl Default for Regs { + fn default() -> Self { + Self { + control: [0; 8], + routes: [0; 128], + global: 0, + errors: 0, + sideband: 0, + } + } +} +pub struct BedrockNi { + regs: Arc>, + error: Mutex>, + delivery: ReentrantMutex<()>, + driven: Mutex>, + reset: Mutex>>, +} +impl Default for BedrockNi { + fn default() -> Self { + Self::new() + } +} +impl BedrockNi { + pub fn new() -> Self { + let regs = Arc::new(Mutex::new(Regs::default())); + crate::locks::register_mutex("bedrock-ni", ®s); + Self { + regs, + error: Mutex::new(None), + delivery: ReentrantMutex::new(()), + driven: Mutex::new(None), + reset: Mutex::new(None), + } + } + pub fn connect_error(&self, cb: Level) { + let _delivery = self.delivery.lock(); + *self.error.lock() = Some(cb); + *self.driven.lock() = None; + self.update_irq(); + } + pub fn connect_local_reset(&self, cb: Arc) { + *self.reset.lock() = Some(cb); + } + pub fn error_level(&self) -> bool { + self.regs.lock().errors & 0x1fffffff000000 != 0 + } + fn update_irq(&self) { + // Serialize the sample and callback, while leaving registers available + // to the callback. Suppress unchanged levels so callbacks can reenter + // register reads that recompute the output. + let _delivery = self.delivery.lock(); + let cb = self.error.lock().clone(); + if let Some(cb) = cb { + let level = self.error_level(); + let changed = { + let mut driven = self.driven.lock(); + let changed = *driven != Some(level); + *driven = Some(level); + changed + }; + if changed { + cb(level); + } + } + } + fn mask(a: u64) -> u64 { + match a { + reg::CONTROL_0 => 1, + reg::CONTROL_1 => 7, + reg::CONTROL_2 => 0x1ff, + reg::CONTROL_3 => 0x7efffffff, + reg::CONTROL_4 => 0xffff, + _ => 0, + } + } +} +impl Resettable for BedrockNi { + fn power_on(&self) { + let mut r = self.regs.lock(); + let sideband = r.sideband; + *r = Regs::default(); + r.sideband = sideband; + drop(r); + self.update_irq(); + } +} +impl BusDevice for BedrockNi { + fn read64(&self, a: u64) -> BusRead64 { + if a & 7 != 0 { + return BusRead64::err(); + } + let mut r = self.regs.lock(); + let v = if Self::mask(a) != 0 { + r.control[a as usize / 8] + } else if (reg::ROUTES..reg::ROUTES_END).contains(&a) { + r.routes[(a as usize - 0x1000) / 8] + } else { + match a { + reg::NODE_ID => 0, + reg::ERROR_STATUS => r.errors, + reg::ERROR_STATUS_UPPER | reg::ERROR_STATUS_LOWER => 0, + reg::SIDE_BAND => r.sideband, + reg::ERROR_CLEAR => { + let v = r.errors; + r.errors = 0; + v + } + reg::GLOBAL_CONTROL => r.global, + _ => { + crate::dlog!(crate::devlog::LogModule::Bedrock, "Bedrock NI unsupported read {a:#x}"); + 0 + } + } + }; + drop(r); + self.update_irq(); + BusRead64::ok(v) + } + fn write64(&self, a: u64, v: u64) -> u32 { + if a & 7 != 0 { + return BUS_ERR; + } + let mut r = self.regs.lock(); + if Self::mask(a) != 0 { + r.control[a as usize / 8] = v & Self::mask(a); + } else if (reg::ROUTES..reg::ROUTES_END).contains(&a) { + r.routes[(a as usize - 0x1000) / 8] = v & 31; + } else { + match a { + reg::LOCAL_RESET => {} + reg::ERROR_STATUS => r.errors = v & ((1 << 53) - 1), + reg::SIDE_BAND => r.sideband = v & 0xfffff, + reg::GLOBAL_CONTROL => r.global = v & 31, + _ => crate::dlog!(crate::devlog::LogModule::Bedrock, "Bedrock NI unsupported or read-only write {a:#x} = {v:#x}"), + } + } + drop(r); + self.update_irq(); + if a == reg::LOCAL_RESET && v & 4 != 0 { + let cb = self.reset.lock().clone(); + if let Some(cb) = cb { + cb(); + } + } + BUS_OK + } +} + +#[cfg(test)] +mod tests { + // Cases ported from tests/qtest/sn1-bedrock-test.c in jamesbraid/qemu-sgi-origami. + use super::*; + use std::sync::atomic::{AtomicU64, Ordering}; + + /// The guest programs route fields and reads the error register to acknowledge faults. + #[test] + fn route_registers_store_fields_and_error_reads_clear_pending_errors() { + let d = Arc::new(BedrockNi::new()); + for (a, mask) in [ + (reg::CONTROL_0, 1), + (reg::CONTROL_1, 7), + (reg::CONTROL_2, 0x1ff), + (reg::CONTROL_3, 0x7efffffff), + (reg::CONTROL_4, 0xffff), + ] { + d.write64(a, !0); + assert_eq!(d.read64(a).data, mask); + } + for a in [reg::ROUTES, reg::ROUTES_END - 8] { + d.write64(a, !0); + assert_eq!(d.read64(a).data, 31); + } + d.write64(reg::GLOBAL_CONTROL, !0); + assert_eq!(d.read64(reg::GLOBAL_CONTROL).data, 31); + let weak = Arc::downgrade(&d); + d.connect_error(Arc::new(move |_| { + let d = weak.upgrade().unwrap(); + let _ = d.read64(reg::SIDE_BAND); + })); + let resets = Arc::new(AtomicU64::new(0)); + let count = resets.clone(); + d.connect_local_reset(Arc::new(move || { + count.fetch_add(1, Ordering::SeqCst); + })); + d.write64(reg::CONTROL_0, 0); + d.write64(reg::LOCAL_RESET, 4); + assert_eq!(resets.load(Ordering::SeqCst), 1); + d.write64(reg::ERROR_STATUS, 0xffffff); + assert!(!d.error_level()); + d.write64(reg::ERROR_STATUS, 1 << 24); + assert!(d.error_level()); + d.read64(reg::ERROR_CLEAR); + assert!(!d.error_level()); + d.write64(reg::SIDE_BAND, !0); + d.power_on(); + assert_eq!(d.read64(reg::SIDE_BAND).data, 0xfffff); + assert_eq!(d.read64(reg::CONTROL_3).data, 0); + } + + /// The PROM's one power-on pass writes a sign-extended port-parameter constant and zero + /// to the error and sideband words; each must read back through its defined bits only. + #[test] + fn power_on_programming_pass_reads_back_through_the_defined_bits() { + let d = BedrockNi::new(); + d.write64(reg::CONTROL_0, 1); + d.write64(reg::CONTROL_1, 0); + d.write64(reg::CONTROL_2, 0x166); + d.write64(reg::CONTROL_3, 0xffff_ffff_a3ff_1990); + d.write64(reg::CONTROL_4, 0xc800); + d.write64(reg::ERROR_STATUS, 0); + d.read64(reg::ERROR_CLEAR); + d.write64(reg::SIDE_BAND, 0); + for (a, want) in [ + (reg::CONTROL_0, 1), + (reg::CONTROL_1, 0), + (reg::CONTROL_2, 0x166), + (reg::CONTROL_3, 0x7_a3ff_1990), + (reg::CONTROL_4, 0xc800), + (reg::ERROR_STATUS, 0), + (reg::SIDE_BAND, 0), + (reg::ERROR_CLEAR, 0), + ] { + assert_eq!(d.read64(a).data, want, "register {a:#x}"); + } + } + + /// Only the local-reset bit of the reset word reaches the node; the other bits must + /// not restart it, and the interface's own registers are left to the board's reset. + #[test] + fn only_the_local_reset_bit_requests_a_node_reset() { + let d = BedrockNi::new(); + let resets = Arc::new(AtomicU64::new(0)); + let count = resets.clone(); + d.connect_local_reset(Arc::new(move || { + count.fetch_add(1, Ordering::SeqCst); + })); + let mut expected = 0; + for enable in 0..2 { + d.write64(reg::CONTROL_0, enable); + for command in 0..8u64 { + d.write64(reg::LOCAL_RESET, command); + expected += u64::from(command & 4 != 0); + assert_eq!(resets.load(Ordering::SeqCst), expected, "command {command:#x}"); + } + } + // With nothing wired to the callback the request does not clear this block. + assert_eq!(d.read64(reg::CONTROL_0).data, 1); + } + + /// A latched port error raises the interrupt line through its interrupting bits only; + /// drivers rely on the line dropping when the status is rewritten or cleared. + #[test] + fn stored_error_status_drives_the_error_line_through_the_interrupting_bits() { + let d = BedrockNi::new(); + let level = Arc::new(AtomicU64::new(0)); + let seen = level.clone(); + d.connect_error(Arc::new(move |high| seen.store(u64::from(high), Ordering::SeqCst))); + for (value, high) in [(1u64 << 30, 1), (0xff, 0), (1 << 52, 1), (0, 0)] { + d.write64(reg::ERROR_STATUS, value); + assert_eq!(level.load(Ordering::SeqCst), high, "status {value:#x}"); + } + // Bit 53 is beyond the register and must not be retained. + d.write64(reg::ERROR_STATUS, 1 << 53); + assert_eq!(d.read64(reg::ERROR_STATUS).data, 0); + } + + /// Reset values of the programmable words are unknown on hardware, so the model clears them; + /// a guest that re-runs its setup after a reset must see zeroes, not stale routes. + #[test] + fn reset_clears_every_programmable_register_and_misaligned_accesses_fault() { + let d = BedrockNi::new(); + let offsets = [ + reg::CONTROL_0, reg::CONTROL_1, reg::CONTROL_2, reg::CONTROL_3, reg::CONTROL_4, + reg::ROUTES, reg::ROUTES_END - 8, reg::GLOBAL_CONTROL, + ]; + for a in offsets { + d.write64(a, !0); + } + // A misaligned write must not land in the neighbouring register. + assert_eq!(d.write64(reg::CONTROL_4 + 1, 0), BUS_ERR); + assert_eq!(d.read64(reg::CONTROL_4 + 1).status, BUS_ERR); + assert_eq!(d.read64(reg::CONTROL_4).data, 0xffff); + d.power_on(); + for a in offsets { + assert_eq!(d.read64(a).data, 0, "register {a:#x}"); + } + } +} diff --git a/src/dev/bedrock_pi.rs b/src/dev/bedrock_pi.rs new file mode 100644 index 000000000..ed93012ac --- /dev/null +++ b/src/dev/bedrock_pi.rs @@ -0,0 +1,791 @@ +//! Bedrock PI exposes processor-slot control, pending interrupts, real-time +//! counters, timers and shared graphics registers. Fuel uses PI0 slot A for +//! its processor; the other processor slot has no device attached. Node hooks +//! carry cache-alias changes, CPU enable and NMI events to the board. +//! +//! # What is approximated here +//! +//! The model accepts Linux's public `SGIrules` key at offset 0x8 for +//! compatibility. Linux documents the literal for Hub I/O, but the available +//! evidence does not establish that this PI offset maps to the same register. +//! Unconnected graphics registers read as zero and are logged as unsupported. +//! +//! Ported from hw/misc/sgi-bedrock-pi.c in jamesbraid/qemu-sgi-origami. + +use crate::dev::bedrock_io::IIO_PROTECT_OVRRD_KEY; +use crate::dev::heart::Clock; +use crate::hptimer::{TimerId, TimerManager, TimerReturn}; +use crate::irq::InterruptController; +use crate::traits::{BusDevice, BusRead32, BusRead64, Device, Resettable, BUS_ERR, BUS_OK}; +use parking_lot::{Mutex, ReentrantMutex}; +use std::sync::{Arc, OnceLock, Weak}; +use std::time::Duration; +const RT_MASK: u64 = (1 << 55) - 1; +const ERROR_MASK: u64 = (1 << 35) - 1; +const MEMORY_ERROR: u64 = 1 << 24; +const REFLECT: u64 = (1 << 54) | (1 << 55) | (1 << 62) | (1 << 63); +const TICK: u64 = 800; +pub const SIZE: u64 = 0x30108; + +pub mod reg { + pub const NODE_REGION: u64 = 0x0; + pub const PROTECTION_KEY: u64 = 0x8; + pub const OFFSET_0010: u64 = 0x10; + pub const OFFSET_0018: u64 = 0x18; + pub const INTERFACE_ID: u64 = 0x20; + pub const CACHED_ALIAS: u64 = 0x28; + pub const OFFSET_0030: u64 = 0x30; + pub const OFFSET_0038: u64 = 0x38; + pub const INTERFACE_SLOT_A: u64 = 0x40; + pub const INTERFACE_SLOT_B: u64 = 0x48; + pub const CPU_A_ENABLE: u64 = 0x50; + pub const CPU_B_ENABLE: u64 = 0x58; + pub const NMI_SLOT_A: u64 = 0x70; + pub const NMI_SLOT_B: u64 = 0x78; + pub const PENDING_MODIFY: u64 = 0x90; + pub const PENDING_STATUS_A: u64 = 0x98; + pub const PENDING_STATUS_B: u64 = 0xa0; + pub const OFFSET_00A8: u64 = 0xa8; + pub const OFFSET_00B0: u64 = 0xb0; + pub const OFFSET_00B8: u64 = 0xb8; + pub const OFFSET_00C0: u64 = 0xc0; + pub const REQUEST_A: u64 = 0xc8; + pub const REQUEST_B: u64 = 0xd0; + pub const PENDING_A_CLEAR: u64 = 0xd8; + pub const PENDING_B_CLEAR: u64 = 0xe0; + pub const OFFSET_00E8: u64 = 0xe8; + pub const OFFSET_00F0: u64 = 0xf0; + pub const RT_COMPARE_A: u64 = 0x108; + pub const RT_COMPARE_B: u64 = 0x110; + pub const PROFILE_COMPARE: u64 = 0x118; + pub const RT_STATUS_A: u64 = 0x120; + pub const RT_STATUS_B: u64 = 0x128; + pub const PROFILE_STATUS_A: u64 = 0x130; + pub const PROFILE_STATUS_B: u64 = 0x138; + pub const RT_ENABLE_A: u64 = 0x140; + pub const RT_ENABLE_B: u64 = 0x148; + pub const PROFILE_ENABLE_A: u64 = 0x150; + pub const PROFILE_ENABLE_B: u64 = 0x158; + pub const PENDING_MODIFY_B: u64 = 0x180; + pub const ERROR_SUMMARY_ALIAS: u64 = 0x3f8; + pub const ERROR_SUMMARY: u64 = 0x400; + pub const ERROR_CAUSE_A: u64 = 0x408; + pub const ERROR_CAUSE_B: u64 = 0x410; + pub const ERROR_ADDRESS_A: u64 = 0x418; + pub const ERROR_ADDRESS_B: u64 = 0x420; + pub const ERROR_CONTROL: u64 = 0x428; + pub const ERROR_CAPTURE_A: u64 = 0x438; + pub const ERROR_CAPTURE_B: u64 = 0x448; + pub const ERROR_CAPTURE_C: u64 = 0x458; + pub const ERROR_CAPTURE_D: u64 = 0x468; + pub const ERROR_SOURCE_A: u64 = 0x490; + pub const ERROR_SOURCE_B: u64 = 0x4c0; + pub const ERROR_DETAIL_A: u64 = 0x4a8; + pub const ERROR_DETAIL_B: u64 = 0x4b0; + pub const ERROR_DETAIL_C: u64 = 0x4b8; + pub const OFFSET_0430: u64 = 0x430; + pub const OFFSET_0440: u64 = 0x440; + pub const OFFSET_0450: u64 = 0x450; + pub const OFFSET_0460: u64 = 0x460; + pub const OFFSET_04C8: u64 = 0x4c8; + pub const OFFSET_04D0: u64 = 0x4d0; + pub const CREDIT_MODE: u64 = 0x20068; + pub const COUNTER: u64 = 0x30100; + pub const GRAPHICS_BASE: u64 = 0x300; + pub const GRAPHICS_END: u64 = 0x350; +} + +#[derive(Clone, Copy, Debug, Default)] +pub struct BedrockPiConfig { + pub interface: u8, + pub request_region: u8, +} +/// Called outside the state lock. Graphics indices 0..9 are the shared PI +/// graphics registers at offsets 0x300..0x348. Disconnected access is observable. +pub trait NodeHooks: Send + Sync { + fn cached_alias_changed(&self, _size: u8) {} + fn rt_compare_written(&self, _interface: u8, _slot: usize) {} + fn rt_counter_read(&self, _interface: u8) {} + fn cpu_enable(&self, _interface: u8, _slot: usize, _enabled: bool) {} + fn nmi(&self, _interface: u8, _slot: usize) {} + fn gfx_read(&self, _index: usize) -> Option { + None + } + fn gfx_write(&self, _index: usize, _value: u64) -> bool { + false + } + fn gfx_reset(&self) {} +} +/// The five registers of each of the two graphics slots, in address order: +/// page, credit counter, bias, interrupt counter and interrupt compare. +const GRAPHICS_SLOT_REGS: usize = 5; +const GRAPHICS_REGS: usize = 2 * GRAPHICS_SLOT_REGS; +/// Each register's writable bits: bit 40 enables a page and bits 39:17 name +/// it; the counters are 12 bits, the interrupt pair 26. +const GRAPHICS_MASKS: [u64; GRAPHICS_SLOT_REGS] = [(1 << 40) | 0xff_fffe_0000, 0xfff, 0xfff, 0x3ff_ffff, 0x3ff_ffff]; + +/// A processor interface's graphics flow-control registers, kept as written +/// with each register's reserved bits dropped. IRIX's graphics driver writes +/// them at attach and tears the board down when the credit counter reads +/// zero. Nothing counts stores or returns credits (no board links to the +/// interface), so the registers hold only what the driver stored. +/// +/// Ported from hw/mips/sgi/pi-gfx.c in jamesbraid/qemu-sgi-origami. +#[derive(Default)] +pub struct GraphicsRegisters(Mutex<[u64; GRAPHICS_REGS]>); +impl NodeHooks for GraphicsRegisters { + fn gfx_read(&self, index: usize) -> Option { + self.0.lock().get(index).copied() + } + fn gfx_write(&self, index: usize, value: u64) -> bool { + match self.0.lock().get_mut(index) { + Some(r) => { + *r = value & GRAPHICS_MASKS[index % GRAPHICS_SLOT_REGS]; + true + } + None => false, + } + } + fn gfx_reset(&self) { + *self.0.lock() = [0; GRAPHICS_REGS]; + } +} +struct State { + w: [u64; 160], + pending: [u64; 2], + reflected: u64, + uart: bool, + base: u64, + epoch: u64, + paused: Option, + running: bool, + rt_due: [u128; 2], + profile_due: u128, + levels: [u8; 256], + rearm: [u64; 128], + credit_mode: u64, +} +impl State { + fn new(now: u64) -> Self { + let mut s = Self { + w: [0; 160], + pending: [0; 2], + reflected: 0, + uart: false, + base: 0, + epoch: now, + paused: Some(now), + running: false, + rt_due: [0; 2], + profile_due: 0, + levels: [0; 256], + rearm: [0; 128], + credit_mode: 0, + }; + s.w[0] = !0; + s.w[2] = !0; + s.w[3] = !0; + s.arm_all(now); + s + } + fn time(&self, now: u64) -> u64 { + self.paused.unwrap_or(now) + } + fn count(&self, now: u64) -> u64 { + self.base + .wrapping_add(self.time(now).saturating_sub(self.epoch) / TICK) + & RT_MASK + } + fn arm_rt(&mut self, slot: usize, now: u64) { + let now = self.time(now); + let ticks = (self.w[0x108 / 8 + slot] + .wrapping_sub(self.count(now)) + .wrapping_sub(1) + & RT_MASK) + + 1; + self.rt_due[slot] = u128::from(now - now.saturating_sub(self.epoch) % TICK) + + u128::from(ticks) * u128::from(TICK); + } + fn arm_profile(&mut self, now: u64) { + let now = self.time(now); + let ticks = (self.w[0x118 / 8] as u32).wrapping_sub(self.count(now) as u32); + let ticks = if ticks == 0 { + 1u64 << 32 + } else { + u64::from(ticks) + }; + self.profile_due = u128::from(now - now.saturating_sub(self.epoch) % TICK) + + u128::from(ticks) * u128::from(TICK); + } + fn arm_all(&mut self, now: u64) { + self.arm_rt(0, now); + self.arm_rt(1, now); + self.arm_profile(now); + } + fn sync(&mut self, now: u64) { + let now = self.time(now); + for slot in 0..2 { + if u128::from(now) >= self.rt_due[slot] { + self.w[0x120 / 8 + slot] = 1; + self.arm_rt(slot, now); + } + } + if u128::from(now) >= self.profile_due { + self.w[0x130 / 8] = 1; + self.w[0x138 / 8] = 1; + self.arm_profile(now); + } + for level in 7..128 { + if self.rearm[level] != 0 && self.rearm[level] <= now { + self.rearm[level] = 0; + if self.levels.iter().any(|&v| usize::from(v) == level + 1) { + self.pending[level / 64] |= 1 << (level % 64); + } + } + } + } + fn pending(&self, bank: usize) -> u64 { + if bank == 1 { + (self.pending[1] & !REFLECT) | self.reflected + } else { + self.pending[0] + | if self.uart { 1 << 4 } else { 0 } + | if self.w[0xc8 / 8] & self.w[0xe8 / 8] != 0 { + 1 << 5 + } else { + 0 + } + | if self.w[0xd0 / 8] & self.w[0xe8 / 8] != 0 { + 1 << 6 + } else { + 0 + } + } + } + fn outputs(&self) -> [bool; 10] { + let mut out = [false; 10]; + for slot in 0..2 { + out[slot * 5] = self.pending(0) & self.w[0xa8 / 8 + slot * 2] != 0; + out[slot * 5 + 1] = self.pending(1) & self.w[0xb0 / 8 + slot * 2] != 0; + out[slot * 5 + 2] = self.w[0x120 / 8 + slot] & self.w[0x140 / 8 + slot] != 0; + out[slot * 5 + 3] = self.w[0x130 / 8 + slot] & self.w[0x150 / 8 + slot] != 0; + out[slot * 5 + 4] = self.w[0x400 / 8] & self.w[0x408 / 8 + slot] != 0; + } + out + } + fn modify(&mut self, value: u64, reissue: bool, now: u64) { + let bank = ((value >> 6) & 1) as usize; + let bit = value & 63; + let mask = 1 << bit; + if bank == 1 && mask & REFLECT != 0 { + return; + } + if value & 0x100 != 0 { + self.pending[bank] |= mask; + } else { + self.pending[bank] &= !mask; + let level = bank * 64 + bit as usize; + if reissue && self.levels.iter().any(|&v| usize::from(v) == level + 1) { + self.rearm[level] = now.saturating_add(1_000_000); + } + } + } +} +pub struct BedrockPi { + cfg: BedrockPiConfig, + clock: Clock, + state: Mutex, + hooks: Mutex>>, + irq: Mutex>>, + timers: OnceLock>, + timer: Mutex>, + timer_pass: Mutex<()>, + dispatch: ReentrantMutex<()>, + driven: Mutex<[Option; 10]>, + weak: Weak, +} +impl BedrockPi { + pub fn new(cfg: BedrockPiConfig, clock: Clock) -> Arc { + assert!(cfg.interface < 2 && cfg.request_region < 64); + Arc::new_cyclic(|weak| Self { + cfg, + state: Mutex::new(State::new(clock())), + clock, + hooks: Mutex::new(None), + irq: Mutex::new(None), + timers: OnceLock::new(), + timer: Mutex::new(None), + timer_pass: Mutex::new(()), + dispatch: ReentrantMutex::new(()), + driven: Mutex::new([None; 10]), + weak: weak.clone(), + }) + } + pub fn set_node_hooks(&self, hooks: Arc) { + let _dispatch = self.dispatch.lock(); + *self.hooks.lock() = Some(hooks); + } + pub fn set_timer_manager(&self, tm: Arc) { + let _ = self.timers.set(tm); + self.schedule(); + } + /// Lines 0..4 are slot A IP2..IP6; lines 5..9 are slot B IP2..IP6. + pub fn set_interrupt_controller(&self, irq: Arc) { + let _dispatch = self.dispatch.lock(); + *self.irq.lock() = Some(irq); + *self.driven.lock() = [None; 10]; + self.drive(); + } + fn hooks(&self) -> Option> { + self.hooks.lock().clone() + } + /// Report an access to a register the model does not implement. + fn note(&self, off: u64, write: bool) { + crate::dlog!(crate::devlog::LogModule::Bedrock, + "Bedrock PI{} unsupported {} at {off:#x}", self.cfg.interface, + if write { "write" } else { "read" }); + } + fn drive(&self) { + // Serialize publication, including reentrant reads by a controller. + // Publish one line at a time: a callback may change another line. + let _dispatch = self.dispatch.lock(); + let irq = self.irq.lock().clone(); + if let Some(irq) = irq { + for line in 0..10 { + let level = self.state.lock().outputs()[line]; + let changed = { + let mut driven = self.driven.lock(); + let changed = driven[line] != Some(level); + driven[line] = Some(level); + changed + }; + if changed { + irq.set_line(line as u32, level); + } + } + } + } + fn schedule(&self) { + let _pass = self.timer_pass.lock(); + let Some(tm) = self.timers.get() else { return }; + let mut timer = self.timer.lock(); + if let Some(id) = timer.take() { + tm.remove(id); + } + let s = self.state.lock(); + if !s.running { + return; + } + let next = s + .rt_due + .iter() + .copied() + .chain([s.profile_due]) + .chain(s.rearm.iter().filter(|&&v| v != 0).map(|&v| u128::from(v))) + .min() + .unwrap(); + let delay = next + .saturating_sub(u128::from((self.clock)())) + .min(1_000_000_000) + .max(1) as u64; + drop(s); + *timer = Some( + tm.add_one_shot(Duration::from_nanos(delay), self.weak.clone(), |weak| { + if let Some(pi) = weak.upgrade() { + { + let _pass = pi.timer_pass.lock(); + let mut s = pi.state.lock(); + if !s.running { + return TimerReturn::Delete; + } + s.sync((pi.clock)()); + } + pi.drive(); + pi.schedule(); + } + TimerReturn::Delete + }), + ); + } + pub fn poll(&self) { + self.state.lock().sync((self.clock)()); + self.drive(); + } + pub fn set_pending(&self, bank: usize, bit: u8) { + if bank < 2 && bit < 64 { + self.state.lock().pending[bank] |= 1 << bit; + self.drive(); + } + } + fn reflected(&self, bit: u8, level: bool) { + let mut s = self.state.lock(); + if level { + s.reflected |= 1 << bit; + } else { + s.reflected &= !(1 << bit); + } + drop(s); + self.drive(); + } + pub fn set_lb_error(&self, level: bool) { + self.reflected(55, level); + } + pub fn set_ni_error(&self, level: bool) { + self.reflected(62, level); + } + pub fn set_xb_error(&self, level: bool) { + self.reflected(54, level); + } + pub fn set_uart_irq(&self, level: bool) { + self.state.lock().uart = level; + self.drive(); + } + pub fn set_memory_uncorrectable(&self) { + self.state.lock().w[0x400 / 8] |= MEMORY_ERROR; + self.drive(); + } + pub fn set_level_source(&self, source: usize, level: u8, active: bool) { + if source < 256 && (!active || (7..=127).contains(&level)) { + self.state.lock().levels[source] = if active { level + 1 } else { 0 }; + self.drive(); + } + } + pub fn io_modify(&self, value: u64, region: u8) -> bool { + let mut s = self.state.lock(); + if region >= 64 || s.w[2] & (1 << region) == 0 { + drop(s); + self.note(0x90, true); + return false; + } + let now = s.time((self.clock)()); + s.modify(value, false, now); + drop(s); + self.drive(); + true + } + fn read(&self, off: u64) -> u64 { + if (reg::GRAPHICS_BASE..reg::GRAPHICS_END).contains(&off) { + if let Some(v) = self + .hooks() + .and_then(|h| h.gfx_read(((off - 0x300) / 8) as usize)) + { + return v; + } + self.note(off, false); + return 0; + } + let mut s = self.state.lock(); + s.sync((self.clock)()); + let value = match off { + reg::INTERFACE_ID => u64::from(self.cfg.interface) << 1, + reg::INTERFACE_SLOT_A => u64::from(self.cfg.interface == 0), + reg::INTERFACE_SLOT_B => 0, + reg::PENDING_STATUS_A => s.pending(0), + reg::PENDING_STATUS_B => s.pending(1), + reg::PENDING_A_CLEAR => s.w[reg::REQUEST_A as usize / 8], + reg::PENDING_B_CLEAR => s.w[reg::REQUEST_B as usize / 8], + reg::ERROR_SUMMARY_ALIAS => s.w[reg::ERROR_SUMMARY as usize / 8], + reg::ERROR_CAPTURE_A | reg::ERROR_CAPTURE_B | reg::ERROR_CAPTURE_C | reg::ERROR_CAPTURE_D => s.w[(off - reg::PROTECTION_KEY) as usize / 8], + reg::CREDIT_MODE => s.credit_mode, + reg::COUNTER => s.count((self.clock)()), + _ if readable(off) => s.w[off as usize / 8], + _ => { + self.note(off, false); + 0 + } + }; + drop(s); + self.drive(); + if off == reg::COUNTER { + if let Some(h) = self.hooks() { + h.rt_counter_read(self.cfg.interface); + } + } + value + } + fn write(&self, off: u64, value: u64) { + // Keep register changes and their node notifications in one order. + // The state lock still drops before callbacks; dispatch is reentrant + // so a node callback can read registers and recompute IRQ outputs. + let _dispatch = self.dispatch.lock(); + if (reg::GRAPHICS_BASE..reg::GRAPHICS_END).contains(&off) { + if !self + .hooks() + .is_some_and(|h| h.gfx_write(((off - 0x300) / 8) as usize, value)) + { + self.note(off, true); + } + return; + } + let now = (self.clock)(); + let mut s = self.state.lock(); + s.sync(now); + let mut alias = None; + let mut compare = None; + let mut enable = None; + let mut nmi = None; + match off { + reg::NODE_REGION | reg::OFFSET_0010 | reg::OFFSET_0018 | reg::OFFSET_00A8 | reg::OFFSET_00B0 | reg::OFFSET_00B8 | reg::OFFSET_00C0 | reg::OFFSET_00E8 | reg::OFFSET_04C8 | reg::OFFSET_04D0 => { + s.w[off as usize / 8] = value + } + reg::PROTECTION_KEY => { + if value == IIO_PROTECT_OVRRD_KEY { + s.w[0] |= 1 << self.cfg.request_region; + } + } + reg::CACHED_ALIAS => { + if s.w[5] != value & 15 { + s.w[5] = value & 15; + alias = Some((value & 15) as u8); + } + } + reg::OFFSET_0030 => s.w[6] = value & 255, + reg::OFFSET_0038 => s.w[7] = value & 0xffffff, + reg::CPU_A_ENABLE | reg::CPU_B_ENABLE => { + let slot = ((off - 0x50) / 8) as usize; + s.w[off as usize / 8] = value & 1; + enable = Some((slot, value & 1 != 0)); + } + reg::NMI_SLOT_A | reg::NMI_SLOT_B => { + if value & 1 != 0 { + nmi = Some(((off - 0x70) / 8) as usize); + } else { + self.note(off, true); + } + } + reg::PENDING_MODIFY | reg::PENDING_MODIFY_B => { + let now = s.time(now); + s.modify(value, true, now); + } + reg::REQUEST_A | reg::REQUEST_B => s.w[off as usize / 8] |= 1 << self.cfg.request_region, + reg::PENDING_A_CLEAR | reg::PENDING_B_CLEAR => s.w[(off - reg::OFFSET_0010) as usize / 8] &= !value, + reg::OFFSET_00F0 => { + s.w[0xf0 / 8] = value & 31; + if value & 31 != 0 { + self.note(off, true); + } + } + reg::RT_COMPARE_A | reg::RT_COMPARE_B => { + let slot = ((off - 0x108) / 8) as usize; + s.w[off as usize / 8] = value & RT_MASK; + s.arm_rt(slot, now); + compare = Some(slot); + } + reg::PROFILE_COMPARE => { + s.w[0x118 / 8] = value & 0xffffffff; + s.arm_profile(now); + } + reg::RT_STATUS_A | reg::RT_STATUS_B | reg::PROFILE_STATUS_A | reg::PROFILE_STATUS_B => s.w[off as usize / 8] = 0, + reg::RT_ENABLE_A | reg::RT_ENABLE_B | reg::PROFILE_ENABLE_A | reg::PROFILE_ENABLE_B => s.w[off as usize / 8] = value & 1, + reg::ERROR_SUMMARY_ALIAS => { + s.w[0x400 / 8] = + (s.w[0x400 / 8] & MEMORY_ERROR) | (value & ERROR_MASK & !MEMORY_ERROR) + } + reg::ERROR_SUMMARY => s.w[reg::ERROR_SUMMARY as usize / 8] &= !(value & ERROR_MASK), + reg::ERROR_CAUSE_A | reg::ERROR_CAUSE_B => s.w[off as usize / 8] = value & ERROR_MASK, + reg::ERROR_ADDRESS_A | reg::ERROR_ADDRESS_B => s.w[off as usize / 8] = value & 0x1fffffff8, + reg::ERROR_CONTROL => { + s.w[0x428 / 8] = value & 15; + if value & 15 != 0 { + self.note(off, true); + } + } + reg::ERROR_CAPTURE_A | reg::ERROR_CAPTURE_B | reg::ERROR_CAPTURE_C | reg::ERROR_CAPTURE_D => { + let slot = usize::from(off >= 0x450); + s.w[0x430 / 8 + slot * 4] = 0; + s.w[0x440 / 8 + slot * 4] = 0; + } + reg::ERROR_SOURCE_A | reg::ERROR_SOURCE_B => s.w[off as usize / 8] = value & 63, + reg::ERROR_DETAIL_A | reg::ERROR_DETAIL_B => s.w[off as usize / 8] = value & 0x1fffff, + reg::ERROR_DETAIL_C => s.w[off as usize / 8] = value & 0xfffff, + reg::CREDIT_MODE => s.credit_mode = value & 1, + reg::COUNTER => { + s.base = value & RT_MASK; + s.epoch = s.time(now); + s.arm_all(now); + } + _ => self.note(off, true), + } + drop(s); + if let Some(h) = self.hooks() { + if let Some(size) = alias { + h.cached_alias_changed(size); + } + if let Some(slot) = compare { + h.rt_compare_written(self.cfg.interface, slot); + } + if let Some((slot, value)) = enable { + h.cpu_enable(self.cfg.interface, slot, value); + } + if let Some(slot) = nmi { + h.nmi(self.cfg.interface, slot); + } + } + self.drive(); + self.schedule(); + } +} +fn readable(off: u64) -> bool { + matches!( + off, + reg::NODE_REGION | reg::OFFSET_0010 + | reg::OFFSET_0018 + | reg::CACHED_ALIAS + | reg::OFFSET_0030 + | reg::OFFSET_0038 + | reg::CPU_A_ENABLE + | reg::CPU_B_ENABLE + | reg::OFFSET_00A8 + | reg::OFFSET_00B0 + | reg::OFFSET_00B8 + | reg::OFFSET_00C0 + | reg::REQUEST_A + | reg::REQUEST_B + | reg::OFFSET_00E8 + | reg::OFFSET_00F0 + | reg::RT_COMPARE_A + | reg::RT_COMPARE_B + | reg::PROFILE_COMPARE + | reg::RT_STATUS_A + | reg::RT_STATUS_B + | reg::PROFILE_STATUS_A + | reg::PROFILE_STATUS_B + | reg::RT_ENABLE_A + | reg::RT_ENABLE_B + | reg::PROFILE_ENABLE_A + | reg::PROFILE_ENABLE_B + | reg::ERROR_SUMMARY + | reg::ERROR_CAUSE_A + | reg::ERROR_CAUSE_B + | reg::ERROR_ADDRESS_A + | reg::ERROR_ADDRESS_B + | reg::ERROR_CONTROL + | reg::OFFSET_0430 + | reg::OFFSET_0440 + | reg::OFFSET_0450 + | reg::OFFSET_0460 + | reg::ERROR_SOURCE_A + | reg::ERROR_DETAIL_A + | reg::ERROR_DETAIL_B + | reg::ERROR_DETAIL_C + | reg::ERROR_SOURCE_B + | reg::OFFSET_04C8 + | reg::OFFSET_04D0 + ) +} +impl BusDevice for BedrockPi { + fn read64(&self, addr: u64) -> BusRead64 { + if addr < SIZE && addr & 7 == 0 { + BusRead64::ok(self.read(addr)) + } else { + self.note(addr, false); + BusRead64::err() + } + } + fn read32(&self, addr: u64) -> BusRead32 { + if addr == reg::COUNTER || addr == reg::COUNTER + 4 { + let value = self.read(reg::COUNTER); + BusRead32::ok(if addr & 4 == 0 { + (value >> 32) as u32 + } else { + value as u32 + }) + } else { + self.note(addr, false); + BusRead32::err() + } + } + fn write64(&self, addr: u64, value: u64) -> u32 { + if addr < SIZE && addr & 7 == 0 { + self.write(addr, value); + BUS_OK + } else { + self.note(addr, true); + BUS_ERR + } + } +} +impl Device for BedrockPi { + fn start(&self) { + let now = (self.clock)(); + let mut s = self.state.lock(); + if s.running { + return; + } + if let Some(paused) = s.paused.take() { + let delta = now.saturating_sub(paused); + s.epoch = s.epoch.saturating_add(delta); + for due in &mut s.rt_due { + *due += u128::from(delta); + } + s.profile_due += u128::from(delta); + for due in &mut s.rearm { + if *due != 0 { + *due = due.saturating_add(delta); + } + } + } + s.running = true; + drop(s); + self.schedule(); + } + fn stop(&self) { + { + let _pass = self.timer_pass.lock(); + let mut s = self.state.lock(); + s.sync((self.clock)()); + s.paused = Some(s.time((self.clock)())); + s.running = false; + if let Some(id) = self.timer.lock().take() { + if let Some(tm) = self.timers.get() { + tm.remove(id); + } + } + } + self.drive(); + } + fn is_running(&self) -> bool { + self.state.lock().running + } + fn step(&self, _cycles: u64) { + self.poll(); + } + fn get_clock(&self) -> u64 { + self.state.lock().count((self.clock)()) + } +} +impl Resettable for BedrockPi { + fn power_on(&self) { + let _dispatch = self.dispatch.lock(); + let now = (self.clock)(); + let mut s = self.state.lock(); + // These controls and captures survive the PI block's local reset. + let old = s.w; + let running = s.running; + let reflected = s.reflected; + let uart = s.uart; + *s = State::new(now); + for off in [ + 0, 0x10, 0x18, 0x28, 0x50, 0x58, 0x400, 0x418, 0x420, 0x428, 0x430, 0x440, 0x450, + 0x460, 0x4c8, 0x4d0, + ] { + s.w[off / 8] = old[off / 8]; + } + s.running = running; + s.paused = if running { None } else { Some(now) }; + s.reflected = reflected; + s.uart = uart; + drop(s); + if let Some(h) = self.hooks() { + h.gfx_reset(); + for slot in 0..2 { + let enabled = self.state.lock().w[0x50 / 8 + slot] != 0; + h.cpu_enable(self.cfg.interface, slot, enabled); + } + } + self.drive(); + self.schedule(); + } +} +#[cfg(test)] +mod tests; diff --git a/src/dev/bedrock_pi/tests.rs b/src/dev/bedrock_pi/tests.rs new file mode 100644 index 000000000..5f684ad11 --- /dev/null +++ b/src/dev/bedrock_pi/tests.rs @@ -0,0 +1,276 @@ +// Cases follow tests/qtest/sn1-bedrock-pi-test.c in jamesbraid/qemu-sgi-origami: +// power-on-values, local-reset, pending-reflection, real-time-comparators +// and profile-counter-wrap. + +use super::*; +use std::sync::atomic::{AtomicU64, Ordering}; +fn setup() -> (Arc, Arc) { + let now = Arc::new(AtomicU64::new(0)); + let clock = now.clone(); + let pi = BedrockPi::new( + BedrockPiConfig::default(), + Arc::new(move || clock.load(Ordering::SeqCst)), + ); + pi.start(); + (pi, now) +} +fn r(pi: &BedrockPi, off: u64) -> u64 { + pi.read64(off).data +} +/// The PROM reads PI slot presence and relies on the documented reset survivors. +#[test] +fn the_pi_reports_interface_state_and_retains_registers_across_reset() { + { + let (pi, _) = setup(); + assert_eq!(r(&pi, reg::INTERFACE_ID), 0); + assert_eq!(r(&pi, reg::INTERFACE_SLOT_A), 1); + assert_eq!(r(&pi, reg::INTERFACE_SLOT_B), 0); + pi.write64(reg::NODE_REGION, 0); + pi.write64(reg::PROTECTION_KEY, 0x53474972756c6573); + assert_eq!(r(&pi, reg::NODE_REGION), 1); + let absent = BedrockPi::new( + BedrockPiConfig { + interface: 1, + request_region: 0, + }, + Arc::new(|| 0), + ); + assert_eq!(r(&absent, reg::INTERFACE_ID), 2); + assert_eq!(r(&absent, reg::INTERFACE_SLOT_A), 0); + assert_eq!(pi.read32(reg::INTERFACE_ID).status, BUS_ERR); + assert_eq!(pi.write32(reg::COUNTER, 0), BUS_ERR); + } + { + let (pi, _) = setup(); + pi.write64(reg::PENDING_MODIFY, 0x10c); + assert_eq!(r(&pi, reg::PENDING_STATUS_A), 1 << 12); + pi.write64(reg::PENDING_MODIFY_B, 12); + assert_eq!(r(&pi, reg::PENDING_STATUS_A), 0); + pi.set_lb_error(true); + pi.write64(reg::PENDING_MODIFY, 64 + 55); + assert_eq!(r(&pi, reg::PENDING_STATUS_B), 1 << 55); + pi.write64(reg::PENDING_MODIFY, 0x100 | 64 | 63); + assert_eq!(r(&pi, reg::PENDING_STATUS_B), 1 << 55); + pi.write64(reg::REQUEST_A, 0); + pi.write64(reg::OFFSET_00E8, 1); + assert_eq!(r(&pi, reg::PENDING_STATUS_A), 1 << 5); + pi.write64(reg::PENDING_A_CLEAR, 1); + assert_eq!(r(&pi, reg::PENDING_STATUS_A), 0); + pi.write64(reg::OFFSET_0010, 0); + assert!(!pi.io_modify(0x108, 0)); + pi.write64(reg::OFFSET_0010, 1); + assert!(pi.io_modify(0x108, 0)); + assert_eq!(r(&pi, reg::PENDING_STATUS_A), 1 << 8); + } + { + let (pi, _) = setup(); + pi.write64(reg::CACHED_ALIAS, 9); + pi.write64(reg::CPU_A_ENABLE, 1); + pi.write64(reg::ERROR_ADDRESS_A, !0); + pi.set_memory_uncorrectable(); + pi.write64(reg::ERROR_SUMMARY_ALIAS, 3); + pi.write64(reg::RT_COMPARE_A, 0x24); + pi.power_on(); + assert_eq!(r(&pi, reg::ERROR_SUMMARY), MEMORY_ERROR | 3); + assert_eq!(r(&pi, reg::CACHED_ALIAS), 9); + assert_eq!(r(&pi, reg::CPU_A_ENABLE), 1); + assert_eq!(r(&pi, reg::ERROR_ADDRESS_A), 0x1fffffff8); + assert_eq!(r(&pi, reg::RT_COMPARE_A), 0); + assert_eq!(r(&pi, reg::COUNTER), 0); + pi.write64(reg::ERROR_SUMMARY_ALIAS, 0); + assert_eq!(r(&pi, reg::ERROR_SUMMARY), MEMORY_ERROR); + pi.write64(reg::ERROR_SUMMARY, MEMORY_ERROR); + assert_eq!(r(&pi, reg::ERROR_SUMMARY), 0); + } +} +/// The guest timer drives interrupts even while the processor is stopped. +/// The processor timer can wrap, pause while stopped, and reissue its guest interrupt. +#[test] +fn the_pi_timer_wraps_pauses_and_reissues_interrupts() { + { + let (pi, now) = setup(); + pi.write64(reg::COUNTER, 0x1_0000_0000); + pi.write64(reg::RT_COMPARE_A, 0x1_0000_0002); + now.store(1599, Ordering::SeqCst); + assert_eq!(r(&pi, reg::RT_STATUS_A), 0); + now.store(1600, Ordering::SeqCst); + assert_eq!(r(&pi, reg::RT_STATUS_A), 1); + assert_eq!(pi.read32(reg::COUNTER).data, 1); + assert_eq!(pi.read32(reg::COUNTER + 4).data, 2); + pi.write64(reg::RT_STATUS_A, !0); + assert_eq!(r(&pi, reg::RT_STATUS_A), 0); + pi.write64(reg::COUNTER, RT_MASK); + pi.write64(reg::PROFILE_COMPARE, 0); + now.store(2400, Ordering::SeqCst); + assert_eq!(r(&pi, reg::COUNTER), 0); + assert_eq!(r(&pi, reg::PROFILE_STATUS_A), 1); + assert_eq!(r(&pi, reg::PROFILE_STATUS_B), 1); + pi.write64(reg::PROFILE_STATUS_A, 0); + assert_eq!(r(&pi, reg::PROFILE_STATUS_A), 0); + assert_eq!(r(&pi, reg::PROFILE_STATUS_B), 1); + } + { + let (pi, now) = setup(); + pi.set_level_source(0, 12, true); + pi.write64(reg::PENDING_MODIFY, 0x10c); + pi.write64(reg::PENDING_MODIFY, 12); + now.store(999999, Ordering::SeqCst); + assert_eq!(r(&pi, reg::PENDING_STATUS_A), 0); + now.store(1000000, Ordering::SeqCst); + assert_eq!(r(&pi, reg::PENDING_STATUS_A), 1 << 12); + pi.write64(reg::COUNTER, 0); + pi.write64(reg::RT_COMPARE_A, 2); + now.store(1000400, Ordering::SeqCst); + pi.stop(); + now.store(2000000, Ordering::SeqCst); + assert_eq!(r(&pi, reg::COUNTER), 0); + pi.start(); + now.store(2001200, Ordering::SeqCst); + assert_eq!(r(&pi, reg::RT_STATUS_A), 1); + } +} +/// An interrupt callback may inspect PI while an output changes. +/// An interrupt callback may inspect PI while an output changes. +#[test] +fn pi_interrupt_outputs_follow_masks_and_allow_reentrant_reads() { + struct Probe { + pi: Weak, + bits: AtomicU64, + } + impl InterruptController for Probe { + fn set_line(&self, line: u32, level: bool) { + if let Some(pi) = self.pi.upgrade() { + let _ = pi.read64(reg::PENDING_STATUS_A); + } + if level { + self.bits.fetch_or(1 << line, Ordering::SeqCst); + } else { + self.bits.fetch_and(!(1 << line), Ordering::SeqCst); + } + } + } + let (pi, now) = setup(); + let probe = Arc::new(Probe { + pi: Arc::downgrade(&pi), + bits: AtomicU64::new(0), + }); + pi.set_interrupt_controller(probe.clone()); + pi.write64(reg::OFFSET_00A8, 1 << 12); + pi.write64(reg::OFFSET_00B0, 1 << 55); + pi.write64(reg::PENDING_MODIFY, 0x10c); + pi.set_lb_error(true); + pi.write64(reg::RT_COMPARE_A, 2); + pi.write64(reg::PROFILE_COMPARE, 2); + pi.write64(reg::RT_ENABLE_A, 1); + pi.write64(reg::PROFILE_ENABLE_A, 1); + pi.write64(reg::ERROR_CAUSE_A, MEMORY_ERROR); + pi.set_memory_uncorrectable(); + now.store(1600, Ordering::SeqCst); + pi.poll(); + assert_eq!(probe.bits.load(Ordering::SeqCst), 31); + pi.write64(reg::RT_STATUS_A, 0); + assert_eq!(probe.bits.load(Ordering::SeqCst), 27); + pi.write64(reg::PENDING_MODIFY, 12); + pi.set_lb_error(false); + pi.write64(reg::PROFILE_STATUS_A, 0); + pi.write64(reg::ERROR_SUMMARY, MEMORY_ERROR); + assert_eq!(probe.bits.load(Ordering::SeqCst), 0); +} +/// A guest can wait for a timer interrupt without polling its compare register. +/// The guest may wait for a timer interrupt without polling the compare register. +#[test] +fn the_timer_manager_delivers_expirations_without_mmio_polling() { + struct Probe(std::sync::mpsc::Sender<()>); + impl InterruptController for Probe { + fn set_line(&self, line: u32, level: bool) { + if line == 2 && level { + self.0.send(()).unwrap(); + } + } + } + let pi = BedrockPi::new(BedrockPiConfig::default(), crate::dev::heart::host_clock()); + let tm = Arc::new(TimerManager::new()); + pi.set_timer_manager(tm.clone()); + let (tx, rx) = std::sync::mpsc::channel(); + pi.set_interrupt_controller(Arc::new(Probe(tx))); + pi.write64(reg::RT_COMPARE_A, 12500); + pi.write64(reg::RT_ENABLE_A, 1); + pi.start(); + rx.recv_timeout(Duration::from_secs(2)).unwrap(); + pi.stop(); + assert!(!pi.is_running()); +} +/// Node callbacks may reenter PI, so concurrent writes must keep their observed order. +/// Node callbacks may reenter PI, so concurrent writes must preserve observed order. +#[test] +fn concurrent_pi_updates_publish_callbacks_in_register_order() { + use std::sync::mpsc; + use std::thread; + + struct Hooks { + pi: Weak, + offset: u64, + first_value: u64, + entered: mpsc::Sender<()>, + release: Mutex>, + observed: AtomicU64, + } + impl Hooks { + fn publish(&self, value: u64) { + // A node may inspect PI registers while processing a change. + let _ = self.pi.upgrade().unwrap().read64(self.offset); + if value == self.first_value { + self.entered.send(()).unwrap(); + self.release.lock().recv().unwrap(); + } + self.observed.store(value, Ordering::SeqCst); + } + } + impl NodeHooks for Hooks { + fn cached_alias_changed(&self, size: u8) { + self.publish(u64::from(size)); + } + fn cpu_enable(&self, _: u8, _: usize, enabled: bool) { + self.publish(u64::from(enabled)); + } + } + + for (offset, first_value, second_value) in [(reg::CACHED_ALIAS, 1, 2), (reg::CPU_A_ENABLE, 1, 0), (reg::CPU_B_ENABLE, 1, 0)] { + let (pi, _) = setup(); + let (entered_tx, entered_rx) = mpsc::channel(); + let (release_tx, release_rx) = mpsc::channel(); + let hooks = Arc::new(Hooks { + pi: Arc::downgrade(&pi), + offset, + first_value, + entered: entered_tx, + release: Mutex::new(release_rx), + observed: AtomicU64::new(99), + }); + pi.set_node_hooks(hooks.clone()); + let first_pi = pi.clone(); + let first = thread::spawn(move || first_pi.write64(offset, first_value)); + entered_rx.recv_timeout(Duration::from_secs(2)).unwrap(); + let (started_tx, started_rx) = mpsc::channel(); + let (done_tx, done_rx) = mpsc::channel(); + let second_pi = pi.clone(); + let second = thread::spawn(move || { + started_tx.send(()).unwrap(); + second_pi.write64(offset, second_value); + done_tx.send(()).unwrap(); + }); + started_rx.recv_timeout(Duration::from_secs(2)).unwrap(); + let premature = done_rx.recv_timeout(Duration::from_millis(100)); + // Always release and join both writers before checking results. + release_tx.send(()).unwrap(); + assert_eq!(first.join().unwrap(), BUS_OK); + second.join().unwrap(); + assert_eq!( + hooks.observed.load(Ordering::SeqCst), + r(&pi, offset), + "offset {offset:#x}" + ); + assert!(matches!(premature, Err(mpsc::RecvTimeoutError::Timeout))); + assert_eq!(r(&pi, offset), second_value); + } +} diff --git a/src/dev/bedrock_xb.rs b/src/dev/bedrock_xb.rs new file mode 100644 index 000000000..6df8e7b1c --- /dev/null +++ b/src/dev/bedrock_xb.rs @@ -0,0 +1,278 @@ +//! Bedrock XB is the node's internal crossbar status block. It records the +//! fabric error queues and lets software service queued errors; the Fuel board +//! connects its outputs to the Bedrock processor interface. +//! +//! # What is approximated here +//! +//! Arbitration timing and diagnostic outputs are not modeled. Unsupported +//! queue reads return zero, while writes are logged and ignored. +//! +//! Ported from hw/misc/sgi-bedrock-xb.c in jamesbraid/qemu-sgi-origami. + +/// Length of the block-relative register aperture. +pub const SIZE: u64 = 0x198; + +pub mod reg { + pub const CONTROL_BASE: u64 = 0x0; + pub const CONTROL_END: u64 = 0x18; + pub const QUEUE_BASE: u64 = 0x20; + pub const QUEUE_END: u64 = 0x88; + pub const FIRST_ERROR: u64 = 0x90; + pub const DEBUG_BASE: u64 = 0x98; + pub const DEBUG_END: u64 = 0xa0; + pub const CLEAR_QUEUE_BASE: u64 = 0x120; + pub const CLEAR_QUEUE_END: u64 = 0x188; + pub const CLEAR_FIRST_ERROR: u64 = 0x190; +} + +use crate::traits::{BusDevice, BusRead64, Resettable, BUS_ERR, BUS_OK}; +use parking_lot::{Mutex, ReentrantMutex}; +use std::sync::Arc; +type Level = Arc; +const MASKS: [u64; 14] = [ + 0x3333, 0x33, 0, 0, 0x3330, 0, 0, 0xf0, 0xfff0, 0, 0, 0, 0x3303, 0x11, +]; +struct Regs { + control: [u64; 4], + queues: [u64; 14], + first: u64, + debug: [u64; 2], + unserviced: u16, +} +impl Default for Regs { + fn default() -> Self { + Self { + control: [0x0008000080000021, 0, 0, 0], + queues: [0; 14], + first: 0, + debug: [0; 2], + unserviced: 0, + } + } +} +pub struct BedrockXb { + regs: Arc>, + error: Mutex>, + delivery: ReentrantMutex<()>, + driven: Mutex>, +} +impl Default for BedrockXb { + fn default() -> Self { + Self::new() + } +} +impl BedrockXb { + pub fn new() -> Self { + let regs = Arc::new(Mutex::new(Regs::default())); + crate::locks::register_mutex("bedrock-xb", ®s); + Self { + regs, + error: Mutex::new(None), + delivery: ReentrantMutex::new(()), + driven: Mutex::new(None), + } + } + pub fn connect_error(&self, cb: Level) { + let _delivery = self.delivery.lock(); + *self.error.lock() = Some(cb); + *self.driven.lock() = None; + self.update_irq(); + } + pub fn error_level(&self) -> bool { + self.regs.lock().first & 0x1000 != 0 + } + fn update_irq(&self) { + // Serialize the sample and callback, while leaving registers available + // to the callback. Suppress unchanged levels so callbacks can reenter + // register reads that recompute the output. + let _delivery = self.delivery.lock(); + let cb = self.error.lock().clone(); + if let Some(cb) = cb { + let level = self.error_level(); + let changed = { + let mut driven = self.driven.lock(); + let changed = *driven != Some(level); + *driven = Some(level); + changed + }; + if changed { + cb(level); + } + } + } +} +impl Resettable for BedrockXb { + fn power_on(&self) { + *self.regs.lock() = Regs::default(); + self.update_irq(); + } +} +impl BusDevice for BedrockXb { + fn read64(&self, a: u64) -> BusRead64 { + if a & 7 != 0 { + return BusRead64::err(); + } + let r = self.regs.lock(); + let v = match a { + reg::CONTROL_BASE..=reg::CONTROL_END => r.control[a as usize / 8], + reg::QUEUE_BASE..=reg::QUEUE_END => { + let n = (a as usize - 0x20) / 8; + if MASKS[n] == 0 { + crate::dlog!(crate::devlog::LogModule::Bedrock, "Bedrock XB unsupported queue {n}"); + } + r.queues[n] + } + reg::FIRST_ERROR => r.first, + reg::DEBUG_BASE | reg::DEBUG_END => r.debug[(a as usize - 0x98) / 8], + _ => { + crate::dlog!(crate::devlog::LogModule::Bedrock, "Bedrock XB unsupported read {a:#x}"); + 0 + } + }; + BusRead64::ok(v) + } + fn write64(&self, a: u64, v: u64) -> u32 { + if a & 7 != 0 { + return BUS_ERR; + } + let mut r = self.regs.lock(); + match a { + reg::CONTROL_BASE..=reg::CONTROL_END => { + let n = a as usize / 8; + r.control[n] = v & [0x000ffffffffffff1, 0xffffffff, 0x7ff, 0xf][n]; + if a == 0x10 && r.control[n] != 0 { + crate::dlog!(crate::devlog::LogModule::Bedrock, "Bedrock XB traffic observation unsupported"); + } + } + reg::QUEUE_BASE..=reg::QUEUE_END => { + let n = (a as usize - 0x20) / 8; + let v = v & MASKS[n]; + if MASKS[n] == 0 { + crate::dlog!(crate::devlog::LogModule::Bedrock, "Bedrock XB unsupported queue {n}"); + } + if v != 0 { + r.queues[n] |= v; + r.unserviced |= 1 << n; + if r.first & 0x1000 == 0 { + r.first = 0x1000 | ((15 - n) as u64) << 8; + } + } + } + reg::CLEAR_QUEUE_BASE..=reg::CLEAR_QUEUE_END => { + let n = (a as usize - 0x120) / 8; + r.queues[n] = 0; + r.unserviced &= !(1 << n); + } + reg::CLEAR_FIRST_ERROR => { + if r.first & 0x1000 != 0 { + let n = 15 - ((r.first >> 8) & 15) as usize; + if r.unserviced & (1 << n) != 0 { + crate::dlog!(crate::devlog::LogModule::Bedrock, "Bedrock XB first error cleared before queue"); + } else { + r.first = 0; + for n in 0..14 { + if r.queues[n] != 0 { + r.first = 0x1000 | ((15 - n) as u64) << 8; + break; + } + } + } + } + } + reg::DEBUG_BASE | reg::DEBUG_END => { + let n = (a as usize - 0x98) / 8; + r.debug[n] = v & [0xffff, 7][n]; + if r.debug[n] != 0 { + crate::dlog!(crate::devlog::LogModule::Bedrock, "Bedrock XB debug output unsupported"); + } + } + _ => crate::dlog!(crate::devlog::LogModule::Bedrock, "Bedrock XB unsupported or read-only write {a:#x} = {v:#x}"), + } + drop(r); + self.update_irq(); + BUS_OK + } +} + +#[cfg(test)] +mod tests { + // Cases follow tests/qtest/sn1-bedrock-xb-test.c in + // jamesbraid/qemu-sgi-origami: control-storage, service-order and + // cold-reset. + use super::*; + + /// The guest sees masked crossbar control words and cleared errors after reset. + #[test] + fn crossbar_control_masks_apply_and_reset_clears_errors() { + let d = BedrockXb::new(); + for (a, mask) in [ + (reg::CONTROL_BASE, 0x000ffffffffffff1), + (reg::CONTROL_BASE + 8, 0xffffffff), + (reg::CONTROL_BASE + 0x10, 0x7ff), + (reg::CONTROL_END, 15), + (reg::DEBUG_BASE, 0xffff), + (reg::DEBUG_END, 7), + ] { + d.write64(a, !0); + assert_eq!(d.read64(a).data, mask); + } + // Keep expected register values independent of the decoder's mask table. + for (address, expected) in [ + (reg::QUEUE_BASE, 0x3333), + (reg::QUEUE_BASE + 8, 0x0033), + (reg::QUEUE_BASE + 0x10, 0), + (reg::QUEUE_BASE + 0x18, 0), + (reg::QUEUE_BASE + 0x20, 0x3330), + (reg::QUEUE_BASE + 0x28, 0), + (reg::QUEUE_BASE + 0x30, 0), + (reg::QUEUE_BASE + 0x38, 0x00f0), + (reg::QUEUE_BASE + 0x40, 0xfff0), + (reg::QUEUE_BASE + 0x48, 0), + (reg::QUEUE_BASE + 0x50, 0), + (reg::QUEUE_BASE + 0x58, 0), + (reg::QUEUE_BASE + 0x60, 0x3303), + (reg::QUEUE_END, 0x0011), + ] { + d.write64(address, !0); + assert_eq!(d.read64(address).data, expected, "register {address:#x}"); + } + d.power_on(); + assert_eq!(d.read64(reg::CONTROL_BASE).data, 0x0008000080000021); + assert!(!d.error_level()); + assert_eq!(d.read32(reg::CONTROL_BASE).status, BUS_ERR); + } + /// The guest services queued crossbar faults in order, one error at a time. + #[test] + fn servicing_one_crossbar_error_exposes_the_next_pending_error() { + { + let d = BedrockXb::new(); + d.write64(reg::QUEUE_BASE, 1); + assert!(d.error_level()); + assert_eq!(d.read64(reg::FIRST_ERROR).data, 0x1f00); + d.write64(reg::CLEAR_FIRST_ERROR, !0); + assert_eq!(d.read64(reg::FIRST_ERROR).data, 0x1f00); + d.write64(reg::CLEAR_QUEUE_END, !0); + d.write64(reg::CLEAR_QUEUE_BASE, 0); + d.write64(reg::CLEAR_FIRST_ERROR, 0); + assert!(!d.error_level()); + d.power_on(); + assert!(!d.error_level()); + } + { + let d = BedrockXb::new(); + d.write64(reg::QUEUE_END, 0x10); + d.write64(reg::QUEUE_BASE, 1); + assert_eq!(d.read64(reg::FIRST_ERROR).data, 0x1200); + d.write64(reg::CLEAR_QUEUE_END, 0); + d.write64(reg::CLEAR_FIRST_ERROR, 0); + assert_eq!(d.read64(reg::FIRST_ERROR).data, 0x1f00); + d.write64(reg::CLEAR_QUEUE_BASE, 0); + d.write64(reg::QUEUE_BASE, 0x10); + d.write64(reg::CLEAR_FIRST_ERROR, 0); + assert_eq!(d.read64(reg::FIRST_ERROR).data, 0x1f00); + d.write64(reg::CLEAR_QUEUE_BASE, 0); + d.write64(reg::CLEAR_FIRST_ERROR, 0); + assert!(!d.error_level()); + } + } +} diff --git a/src/dev/bridge.rs b/src/dev/bridge.rs index 46ed7c255..7d6b64b8d 100644 --- a/src/dev/bridge.rs +++ b/src/dev/bridge.rs @@ -15,6 +15,7 @@ //! | 0x010000 | translation table, 128 eight-byte entries | //! | 0x011000 | the same entries' lower halves, read only | //! | 0x020000 | type 0 configuration space, 4 KB per device | +//! | 0x028000 | type 1 configuration space, 4 KB | //! | 0x080000 | external SSRAM (unpopulated) | //! | 0x200000 | device windows: two of 2 MB, then six of 1 MB | //! | 0xc00000 | flash, two 2 MB chip selects on a 16-bit port | @@ -31,8 +32,12 @@ //! errors are never raised. //! * The translation table is the chip's internal 128 entries; no external //! SRAM extends it. -//! * Processor accesses to the 2 GB long window (the translated PIO window -//! at 0x4000_0000) are not modelled. +//! * Through the widget's 2 GB window the first 16 MB is the small window +//! and 0x4000_0000 on is the 32-bit alias of PCI memory (specification +//! p. 52); the rest of that window is a bus error. The 64-bit alias at +//! 0x8000_0000 and PCI I/O at 4 GB, which only the 64 GB window reaches, +//! are not modelled. Which device a master time-out through the alias +//! names in the error capture is not in the specification; iris names 0. //! * A function's DMA address at or above 4 GiB goes through the 64-bit //! direct window: bits 63:60 name the widget and the low 48 bits pass //! through. PCI-local space (below 0x4000_0000), which would reach @@ -53,8 +58,11 @@ use std::sync::{Arc, Weak}; use parking_lot::{Mutex, RwLock}; use crate::dev::w1; -use crate::irq::{InterruptController, IrqLine}; -use crate::pci::{bar_runs, BarAt, BarRun, BarRunSink, PciBar, PciBusDevice, PciDma, PciDmaStatus, PciLanes, Space}; +use crate::devlog::LogModule; +use crate::irq::InterruptController; +use crate::pci::BarMapping; +use crate::pci::{CpuWindow, DevFn, IntxRouting, PciBus, PciDma, PciDmaStatus, PciHost, Space}; +use crate::pci::{BarDevices, ByteOrder}; use crate::snapshot::{field, field_u32, field_u32s, field_u64, field_u64_vec, field_u8, field_u8s, StateTable}; use crate::traits::{BusDevice, BusRead16, BusRead32, BusRead64, BusRead8, Saveable, BUS_ERR, BUS_OK}; use crate::xtalk::Upstream; @@ -110,17 +118,68 @@ pub mod reg { pub const RRB_CLEAR: u32 = 0x298; } +/// A register's name for the `bridge` log, from its slot base. +fn reg_name(base: u32) -> String { + use reg::*; + let name = match base { + IDENTIFICATION => "identification", + STATUS => "status", + ERR_UPPER => "error upper", + ERR_LOWER => "error lower", + CONTROL => "control", + REQUEST_TIMEOUT => "request timeout", + INT_DEST_UPPER => "interrupt destination upper", + INT_DEST_LOWER => "interrupt destination lower", + ERR_CMD_WORD => "error command word", + LINK_CONFIG => "link config", + TARGET_FLUSH => "target flush", + DIRECT_MAP => "direct map", + ARBITRATION => "arbitration", + NIC => "nic", + BUS_TIMEOUT => "bus timeout", + PCI_TYPE1_CONFIG => "type 1 config", + PCI_ERR_UPPER => "pci error upper", + PCI_ERR_LOWER => "pci error lower", + INT_STATUS => "interrupt status", + INT_ENABLE => "interrupt enable", + INT_RESET => "interrupt reset", + INT_MODE => "interrupt mode", + INT_DEVICE => "interrupt device", + HOST_ERR_FIELD => "host error field", + RRB_EVEN => "rrb even", + RRB_ODD => "rrb odd", + RRB_STATUS => "rrb status", + RRB_CLEAR => "rrb clear", + 0x130..=0x168 => return format!("interrupt host {}", (base - 0x130) / 8), + 0x200..=0x238 => return format!("device {}", (base - 0x200) / 8), + 0x240..=0x278 => return format!("device {} flush", (base - 0x240) / 8), + _ => "", + }; + format!("{base:#05x} {name}") +} + /// Window geometry, as offsets in the small window. pub const ATE_BASE: u32 = 0x01_0000; pub const ATE_LOW_BASE: u32 = 0x01_1000; pub const CFG0_BASE: u32 = 0x02_0000; pub const CFG0_SIZE: u32 = 8 * 0x1000; +pub const TYPE1_BASE: u32 = 0x02_8000; +pub const TYPE1_SIZE: u32 = 0x1000; pub const SSRAM_BASE: u32 = 0x08_0000; pub const SSRAM_SIZE: u32 = 0x08_0000; pub const DEVICE_WINDOW_BASE: u32 = 0x20_0000; pub const FLASH_BASE: u32 = 0xc0_0000; pub const FLASH_SELECT_SIZE: u32 = 0x20_0000; pub const DEVICES: usize = 8; +/// The 32-bit alias of PCI memory space, as an offset in the widget's XIO +/// space: PCI address 0 on (specification p. 52). +pub const PIO32_ALIAS: u32 = 0x4000_0000; +const PIO32_ALIAS_SIZE: u32 = 0x4000_0000; +/// The widget's 2 GB window, and the CPU window index the alias takes +/// after the eight device windows. +pub const LONG_WINDOW_SIZE: u64 = 0x8000_0000; +const LONG_MASK: u64 = LONG_WINDOW_SIZE - 1; +const ALIAS_WINDOW: usize = DEVICES; pub const PINS: usize = 8; pub const ATE_ENTRIES: usize = 128; @@ -167,7 +226,23 @@ const STATUS_PCI_MODE: u32 = 1 << 5; pub const CONTROL_FLASH_WRITE_EN: u32 = 1 << 31; pub const CONTROL_MEM_SWAP: u32 = 1 << 22; pub const CONTROL_PAGE_16K: u32 = 1 << 21; -const CONTROL_RESET: u32 = (1 << 30) | (1 << 29) | (1 << 28) | (0xf << 24) | (3 << 12) | (1 << 9) | (0x1f << 4) | 0xf; +/// Control 27:24, RST_PIN_N(3:0): the four PCI reset pins, active low, all +/// high (deasserted) at reset. Established: the bits, their reset value and +/// that a 1 drives the pin high (Bridge ASIC specification rev 4.0, Control +/// register; the pin list names four `BUS_RSTN` outputs, "PCI Bus reset +/// signals"). +/// +/// Which device each pin serves is not documented. The O2 Device Driver +/// Programming Guide (007-3443-002, pciio(D3), p. 550) describes +/// `pciio_reset()` as activating the reset line of one card "without +/// affecting other devices on the PCI bus", so a pin is taken to reset one +/// device. Pin `n` resets the functions at PCI device `n` and no others: the +/// pin-to-device wiring is inferred, from the pin count and the driver +/// interface alone. A write that takes a pin from 1 to 0 is logged. A core +/// reset and a link reset do not pulse the pins, because the specification +/// does not say that either does. +const CONTROL_RST_PINS: u32 = 0xf << 24; +const CONTROL_RESET: u32 = (1 << 30) | (1 << 29) | (1 << 28) | CONTROL_RST_PINS | (3 << 12) | (1 << 9) | (0x1f << 4) | 0xf; /// Two strobes that clear the link retry counters and always read zero. const CONTROL_RETRY_CLEAR: u32 = 0x3 << 10; @@ -322,12 +397,8 @@ struct State { /// Latched error bits of the interrupt status register (31:8). int_error: u32, /// The eight pins' driven levels, which the status register reports. - /// They follow the level and never latch. + /// They follow the level the bus combines for each pin and never latch. pin_level: u8, - pci_pin_level: u8, - gpio_pin_level: u8, - /// Each slot's INTA..INTD, as `slot * 4 + pin - 1`. - pci_lines: [bool; DEVICES * 4], /// Slots whose data phases are 64 bits wide. dma_64bit: u8, /// The transfer the last DMA translation served, for error capture. @@ -386,50 +457,23 @@ pub struct Bridge { /// The host chips this Bridge's transfers and interrupt packets can go /// to, by the widget number the guest programs. up: Upstream, - slots: RwLock<[Option>; DEVICES]>, + /// The functions on the Bridge's PCI bus, their BAR windows and their + /// interrupt pins. + bus: PciBus, + /// This Bridge, for the DMA a function is given: it must not keep the + /// Bridge alive. + me: Weak, flash: RwLock<[Option; 2]>, flash_strap: AtomicBool, /// The silicon revision in the identification word, bits 31:28. revision: AtomicU32, + /// The devices made for BAR mappings, once each. + bar_devices: BarDevices, /// The one-wire net behind the NIC register; none fitted reads as an /// empty bus. microlan: Mutex>, /// Whether the Crossbow has a live link to a widget; default the hosts. link_alive: RwLock bool + Send + Sync>>, - /// Where the runs of the device windows a BAR decodes whole go (the - /// machine maps them over this widget's window), and the table entries - /// made for them, so one configuration maps to one entry. - bar_sink: RwLock>, - bar_entries: Mutex>>, -} - -/// A function's INTA..INTD, as a level into the Bridge: the line is -/// `slot * 4 + pin - 1`. -struct PciPin { - bridge: Weak, -} - -impl InterruptController for PciPin { - fn set_line(&self, line: u32, level: bool) { - if let Some(b) = self.bridge.upgrade() { - b.set_pci_line(line as usize, level); - } - } -} - -/// One of the chip's eight pins driven directly (the GPIO inputs that -/// share them with PCI), holding the Bridge weakly so the device that -/// holds the line does not keep it alive. -struct WirePin { - bridge: Weak, -} - -impl InterruptController for WirePin { - fn set_line(&self, line: u32, level: bool) { - if let Some(b) = self.bridge.upgrade() { - b.set_wire_irq(line, level); - } - } } /// A function's way to memory, tagged with its slot: the swap bits that @@ -484,35 +528,40 @@ impl Placement { impl Bridge { /// A Bridge that masters memory and interrupts the host chips of `up`, /// each addressed as it states and chosen by the widget number the guest - /// programs. Registers start at their power-up values. - pub fn new(up: Upstream) -> Arc { + /// programs, its PCI bus wired as `wiring`. Registers start at their + /// power-up values. + pub fn new(up: Upstream, wiring: &'static IntxRouting) -> Arc { let mut st = State { reg: [0; REG_SLOTS], ate: [0; ATE_ENTRIES], int_error: 0, pin_level: 0, - pci_pin_level: 0, - gpio_pin_level: 0, - pci_lines: [false; DEVICES * 4], dma_64bit: 0, dma_pci_address: 0, dma_device: 0, }; st.reset_core(false, REVISION); let hosts = (0..16u8).filter(|&w| up.get(w).is_some()).fold(0u16, |m, w| m | 1 << w); - Arc::new(Self { + let this = Arc::new_cyclic(|me: &Weak| Self { st: Mutex::new(st), irq_order: Mutex::new(()), up, - slots: RwLock::new(Default::default()), + bus: PciBus::new(0, wiring, me.clone()), + me: me.clone(), flash: RwLock::new(Default::default()), flash_strap: AtomicBool::new(false), revision: AtomicU32::new(REVISION), microlan: Mutex::new(None), + bar_devices: Default::default(), link_alive: RwLock::new(Box::new(move |w| hosts.checked_shr(w as u32).is_some_and(|m| m & 1 != 0))), - bar_sink: RwLock::new(None), - bar_entries: Mutex::new(Default::default()), - }) + }); + this.bus.set_host_windows(&|| this.windows()); + this + } + + /// The functions on the Bridge's PCI bus. + pub fn bus(&self) -> &PciBus { + &self.bus } pub fn register_locks(self: &Arc) { @@ -538,72 +587,68 @@ impl Bridge { ret } - /// One of the chip's pins as a line a device can hold: the level drives - /// the pin directly, sharing it with whatever PCI functions are wired - /// to it (IOC3's SuperIO interrupt is `pin_line(4)`). The line holds the - /// Bridge weakly. - pub fn pin_line(self: &Arc, pin: u8) -> IrqLine { - IrqLine::new(Arc::new(WirePin { bridge: Arc::downgrade(self) }), pin as u32) - } - - /// Put a function in `slot` (0..8), give it its way to memory and wire - /// its interrupt pin (INTA..INTD per its header) to the Bridge's pins. - pub fn attach(self: &Arc, slot: u8, dev: Arc) { - let slot = slot as usize; - assert!(slot < DEVICES, "Bridge has {DEVICES} slots"); - dev.connect_dma(Arc::new(SlotDma { bridge: Arc::downgrade(self), slot })); - let pin = dev.config().interrupt_pin(); - if (1..=4).contains(&pin) { - let router = Arc::new(PciPin { bridge: Arc::downgrade(self) }); - dev.connect_interrupt(IrqLine::new(router, (slot * 4 + pin as usize - 1) as u32)); - } - self.slots.write()[slot] = Some(dev); - self.publish_bars(); - } - - /// Send the runs of the device windows that one BAR decodes whole to - /// `sink`, now and whenever a Device(x) register, a configuration - /// header or a reset changes them. Each run is a [`PciBar`] offset in - /// this widget's window: the machine maps it over the window, so the - /// processor reaches the function with no search of the bus and no - /// Bridge lock. Everything else in a window (pages a BAR covers in - /// part, an unclaimed address and its master time-out) still comes here. - pub fn on_bar_runs(&self, sink: BarRunSink) { - *self.bar_sink.write() = Some(sink); - self.publish_bars(); - } - - fn publish_bars(&self) { - let sink = self.bar_sink.read(); - let Some(sink) = sink.as_ref() else { return }; - let regs: Vec = { - let st = self.st.lock(); - (0..DEVICES as u32).map(|s| st.r(reg::device(s))).collect() - }; - let funcs: Vec<(usize, Arc)> = - self.slots.read().iter().enumerate().filter_map(|(i, d)| Some((i, d.clone()?))).collect(); - let mut runs: Vec = Vec::new(); - for (s, &d) in regs.iter().enumerate() { - let base = (d & DEV_WINDOW_BASE_MASK) << 20; - let width = if d & DEV_64BIT_DEVICE != 0 { 8 } else { 4 }; - let swap = d & DEV_SWAP_PROGRAMMED != 0; - let space = if d & DEV_MEMORY_SPACE != 0 { Space::Mem } else { Space::Io }; - let (off, size) = (device_window_offset(s as u32) as u64, device_window_size(s as u32) as u64); - runs.extend(bar_runs(&funcs, space, off, base, size, |b| self.bar_entry(b, width, swap))); - } - sink(runs); - } - - fn bar_entry(&self, b: BarAt, width: usize, swap: bool) -> Arc { - let key = (b.slot, b.bar, width, swap, b.cpu_base); - self.bar_entries - .lock() - .entry(key) - .or_insert_with(|| { - let lanes = Arc::new(Lanes { width, swap }); - Arc::new(PciBar::new(lanes, b.dev.clone(), b.bar, WINDOW_MASK as u64, b.cpu_base)) + /// The eight device windows as the Device(x) registers have them: each + /// opens onto PCI memory or I/O space at the base its register names. + /// After them, the 32-bit alias of PCI memory in the 2 GB window. + fn windows(&self) -> Vec { + let st = self.st.lock(); + let alias = CpuWindow { space: Space::Mem, cpu: PIO32_ALIAS as u64, pci: 0, len: PIO32_ALIAS_SIZE as u64, mask: LONG_MASK }; + (0..DEVICES as u32) + .map(|s| { + let d = st.r(reg::device(s)); + CpuWindow { + space: if d & DEV_MEMORY_SPACE != 0 { Space::Mem } else { Space::Io }, + cpu: device_window_offset(s) as u64, + pci: (d & DEV_WINDOW_BASE_MASK) << 20, + len: device_window_size(s) as u64, + mask: WINDOW_MASK as u64, + } }) - .clone() + .chain(std::iter::once(alias)) + .collect() + } + + /// How a CPU access to window `window` reaches a function, from its + /// Device(x) register: the programmed swap gives a byte stream, otherwise + /// the device's width gives word values. + /// Through the alias, Control's memory swap bit decides, as the + /// programmed swap does for a window, and the device counts as 32-bit. + fn order(&self, window: usize) -> ByteOrder { + let st = self.st.lock(); + if window == ALIAS_WINDOW { + return Self::alias_order(st.r(reg::CONTROL)); + } + Self::order_of(st.r(reg::device(window as u32))) + } + + fn alias_order(control: u32) -> ByteOrder { + if control & CONTROL_MEM_SWAP != 0 { ByteOrder::Stream } else { ByteOrder::values(4) } + } + + fn order_of(device: u32) -> ByteOrder { + if device & DEV_SWAP_PROGRAMMED != 0 { + ByteOrder::Stream + } else { + ByteOrder::values(if device & DEV_64BIT_DEVICE != 0 { 8 } else { 4 }) + } + } + + /// The bus device the machine maps over the part of the Bridge's window + /// that `run` covers: the function's BAR through the order the window's + /// Device(x) register sets, so the processor reaches the function with no + /// search of the bus and no Bridge lock. The run is in the widget's small + /// window, so a mapped device sees offsets in it. The same configuration + /// gives back the same device, so updating the mappings does not make the + /// physical bus keep a new one. A run in the alias is in the 2 GB window, + /// and its device sees offsets in that. + pub fn bar_device(&self, run: &BarMapping) -> Arc { + let mask = if run.window as usize == ALIAS_WINDOW { LONG_MASK } else { WINDOW_MASK as u64 }; + self.bar_devices.get(run, mask, self.order(run.window as usize)) + } + + /// What the machine maps over the widget's 2 GB window. + pub fn long_window(self: &Arc) -> Arc { + Arc::new(LongWindow(self.clone())) } /// Fit a flash part on chip select 0. `part` is addressed from the @@ -670,43 +715,22 @@ impl Bridge { st.dma_64bit = (st.dma_64bit & !(1 << slot)) | ((wide as u8) << slot); } - /// Drive one of the chip's directly wired interrupt inputs (the GPIO - /// pins), which share the eight pins with PCI. `pin` is 0..8; anything - /// else is not a pin. - pub fn set_wire_irq(&self, pin: u32, level: bool) { - if pin as usize >= PINS { - return; - } - self.irq_locked(|st, p| { - st.gpio_pin_level = (st.gpio_pin_level & !(1 << pin)) | ((level as u8) << pin); - self.update_pin(st, p, pin as usize); - }); - } - /// Reset the Bridge's own state: the register file, the translation /// table and the latched errors return to power-up. The interrupt /// inputs keep their levels, since the functions holding them are not - /// in this reset and would never raise another edge; the pins are - /// re-evaluated against the cleared enables. + /// in this reset and would never raise another edge. pub fn reset(&self) { let strap = self.flash_strap.load(Ordering::Acquire); let revision = self.revision.load(Ordering::Acquire); - let mut st = self.st.lock(); - st.reset_core(strap, revision); - st.pin_level = st.pci_pin_level | st.gpio_pin_level; - drop(st); - self.publish_bars(); + self.st.lock().reset_core(strap, revision); + self.bus.set_host_windows(&|| self.windows()); } - /// The machine's reset reaching the PCI side: RST# returns every - /// function's configuration header to power-up, so none decodes until - /// firmware assigns its BARs again. The functions' own state is theirs - /// to reset. + /// The machine's reset reaching the PCI side: RST# on the bus. Every + /// function's configuration header returns to power-up, so none decodes + /// until firmware assigns its BARs again. pub fn reset_pci(&self) { - for d in self.slots.read().iter().flatten() { - d.config().reset(); - } - self.publish_bars(); + self.bus.reset(); } /// The Crossbow's link reset arriving. IRIX saves the control and link @@ -733,36 +757,17 @@ impl Bridge { // ---- interrupts ---- - /// Pins A and C reach the pin numbered as the slot, B and D that - /// number with bit two flipped. - fn pin_of(slot: usize, pin0: usize) -> usize { - if pin0 & 1 != 0 { - slot ^ 4 - } else { - slot - } - } - - fn set_pci_line(&self, line: usize, level: bool) { - if line >= DEVICES * 4 { - return; - } - self.irq_locked(|st, p| { - st.pci_lines[line] = level; - // Several functions can share a pin: the pin is the OR of its lines. - let pin = Self::pin_of(line / 4, line % 4); - let any = (0..DEVICES * 4).any(|l| st.pci_lines[l] && Self::pin_of(l / 4, l % 4) == pin); - st.pci_pin_level = (st.pci_pin_level & !(1 << pin)) | ((any as u8) << pin); - self.update_pin(st, p, pin); - }); - } - - fn update_pin(&self, st: &mut State, p: &mut Packets, pin: usize) { - let level = (st.pci_pin_level | st.gpio_pin_level) >> pin & 1 != 0; + /// Pin `pin` is now at `level`, as the bus combined every function and + /// board wire routed to it. The chip sends on a transition, not while a + /// level is held. + fn update_pin(&self, st: &mut State, p: &mut Packets, pin: usize, level: bool) { let was = st.pin_level >> pin & 1 != 0; let enabled = st.r(reg::INT_ENABLE) >> pin & 1 != 0; + if level != was { + crate::dlog!(LogModule::Bridge, "bridge: pin {pin} {}{}", if level { "rose" } else { "fell" }, + if enabled { "" } else { " (not enabled)" }); + } st.pin_level = (st.pin_level & !(1 << pin)) | ((level as u8) << pin); - // The chip sends on a transition, not while a level is held. if level && !was && enabled { self.send_interrupt(st, p, pin, true); } else if !level && was && enabled && st.r(reg::INT_MODE) >> pin & 1 != 0 { @@ -790,7 +795,10 @@ impl Bridge { let address = (dest.address & !(0x3ffu64 << 38)) | field << 38; let vector = (host & INT_HOST_FIELD) as u8; match self.up.get(dest.widget) { - Some(h) if h.host.accepts(dest.widget, address, vector) => p.push((dest.widget, vector, set)), + Some(h) if h.host.accepts(dest.widget, address, vector) => { + crate::dlog!(LogModule::Bridge, "bridge: pin {pin} {} vector {vector} of widget {}", if set { "sets" } else { "clears" }, dest.widget); + p.push((dest.widget, vector, set)); + } Some(_) => log::debug!("bridge: interrupt packet to widget {} address {address:#x} vector {vector} is not one its host takes; dropped", dest.widget), None => log::debug!("bridge: interrupt packet to widget {}, which is not a host; dropped", dest.widget), } @@ -885,7 +893,8 @@ impl Bridge { return false; } let val = val as u32; - self.irq_locked(|st, p| { + let mut asserted = 0u32; + let ok = self.irq_locked(|st, p| { let Some(info) = (if off % 8 == SLOT_DATA { reg_info(off - SLOT_DATA) } else { None }) else { self.invalid_address(st, p, off, true); return true; @@ -916,6 +925,11 @@ impl Bridge { let old = st.r(base); let new = (old & !info.writable) | (val & info.writable); st.set_r(base, new); + // Control 27:24 going from 1 to 0 asserts RST# (see + // CONTROL_RST_PINS). + if base == reg::CONTROL { + asserted = (old & !new & CONTROL_RST_PINS) >> 24; + } if base == reg::NIC { self.onewire_slot(st, val); } else if base == reg::INT_ENABLE { @@ -930,7 +944,13 @@ impl Bridge { } } true - }) + }); + // With the state lock dropped: the functions reset themselves. + for pin in (0..4).filter(|pin| asserted >> pin & 1 != 0) { + log::warn!("bridge: Control drops reset pin {pin}: resetting PCI device {pin} (pin wiring is inferred)"); + self.bus.reset_device(pin); + } + ok } /// One bit slot, reported finished inside the write the guest polls after. @@ -983,36 +1003,91 @@ impl Bridge { /// The function a type 0 configuration offset selects, and the corrected /// register byte offset within it. - fn cfg_target(&self, off: u32, size: usize) -> (usize, u32, Option>) { - let slot = ((off >> 12) & 7) as usize; - let func = (off >> 8) & 7; - let reg = Self::lane_correct(off & 0xff, size); - let dev = if func == 0 { self.slots.read()[slot].clone() } else { None }; - (slot, reg, dev) + fn cfg_target(off: u32, size: usize) -> (DevFn, u32) { + let at = DevFn { device: ((off >> 12) & 7) as u8, function: ((off >> 8) & 7) as u8 }; + (at, Self::lane_correct(off & 0xff, size)) } /// No device drove device select: a master time-out. The cycle's PCI /// address is the device's ID select, one line per device from line 24, /// then the register's word. - fn config_master_timeout(&self, slot: usize, reg: u32) { - self.irq_locked(|st, p| { - self.pci_error(st, p, INT_PCI_MASTER_TIMEOUT, slot as u32, (1u64 << (24 + slot)) | (reg & 0x7fc) as u64) - }); + fn config_master_timeout(&self, slot: u8, reg: u32) { + self.config_error(slot as u32, (1u64 << (24 + slot)) | (reg & 0x7fc) as u64); + } + + fn config_error(&self, device: u32, pci_address: u64) { + self.irq_locked(|st, p| self.pci_error(st, p, INT_PCI_MASTER_TIMEOUT, device, pci_address)); + } + + /// The bus and function a type 1 configuration offset selects, and the + /// corrected register byte offset. The PCI Type 1 Configuration register + /// names the bus (bits 23:16) and device (bits 15:11); the aperture + /// offset gives the function and register (specification tables 26 and + /// 43). A type 1 cycle on the Bridge's bus is claimed only by a + /// PCI-to-PCI bridge whose secondary to subordinate range holds the bus + /// number, so the bus is None when none does. The last element is the + /// cycle's PCI address (type 1 format, AD[1:0] = 01), for the error + /// capture. + fn type1_target(&self, off: u32, size: usize) -> (Option, DevFn, u32, u64) { + let sel = self.st.lock().r(reg::PCI_TYPE1_CONFIG); + let bus_number = (sel >> 16) as u8; + let at = DevFn { device: ((sel >> 11) & 0x1f) as u8, function: ((off >> 8) & 7) as u8 }; + let reg = Self::lane_correct(off & 0xff, size); + let address = (sel & TYPE1_WRITABLE) | (at.function as u32) << 8 | (reg & 0xfc) | 1; + (self.bus.type1_target(bus_number), at, reg, address as u64) + } + + fn type1_read(&self, off: u32, size: usize) -> Option { + if size > 4 { + return None; + } + let (bus, at, reg, address) = self.type1_target(off, size); + let read = bus.and_then(|b| b.config_read(at, reg as usize, size)); + if read.is_none() { + // Which device the capture names for a type 1 cycle is not in + // the specification; the device number the cycle carried. + self.config_error(at.device as u32, address); + } + read.map(u64::from) + } + + fn type1_write(&self, off: u32, size: usize, val: u64) -> bool { + if size > 4 { + return false; + } + let (bus, at, reg, address) = self.type1_target(off, size); + if !bus.is_some_and(|b| b.config_write(at, reg as usize, size, val as u32)) { + self.config_error(at.device as u32, address); + } + true + } + + fn type1_fault(&self, off: u32, size: usize) -> Option { + if size > 4 { + return None; + } + let (bus, _, _, address) = self.type1_target(off, size); + let n = address >> 16 & 0xff; + match bus { + None => Some(format!("PCI type 1 config master timeout: no bridge reaches bus {n}")), + // Behind a bridge an unclaimed cycle master-aborts there and the + // Bridge sees it complete. + Some(_) => None, + } } fn cfg_read(&self, off: u32, size: usize) -> Option { if size > 4 { return None; } - let (slot, reg, dev) = self.cfg_target(off, size); - let Some(dev) = dev else { - self.config_master_timeout(slot, reg); + let (at, reg) = Self::cfg_target(off, size); + let read = self.bus.config_read(at, reg as usize, size); + if read.is_none() { + self.config_master_timeout(at.device, reg); // An unclaimed read fails back to the requester: guests read // empty slots only inside a bus-error guard. - return None; - }; - let dword = dev.config_dword((reg >> 2) as usize) as u64; - Some((dword >> (8 * (reg & 3))) & (u64::MAX >> (64 - 8 * size))) + } + read.map(u64::from) } /// Why a configuration access failed, when the cause is that nothing @@ -1023,12 +1098,12 @@ impl Bridge { if size > 4 { return None; } - let (slot, _, dev) = self.cfg_target(off, size); - if dev.is_some() { + let (at, _) = Self::cfg_target(off, size); + if self.bus.function(at).is_some() { return None; } - let func = (off >> 8) & 7; - Some(if self.slots.read()[slot].is_none() { + let DevFn { device: slot, function: func } = at; + Some(if !self.bus.occupied(slot) { format!("PCI config master timeout: empty slot {slot} function {func}") } else { format!("PCI config master timeout: slot {slot} has no function {func}") @@ -1040,121 +1115,69 @@ impl Bridge { if size > 4 { return false; } - let (slot, reg, dev) = self.cfg_target(off, size); - let Some(dev) = dev else { - self.config_master_timeout(slot, reg); - return true; - }; - let cfg = dev.config(); - let i = (reg >> 2) as usize; - let shift = 8 * (reg & 3); - let mask = (u64::MAX >> (64 - 8 * size)) << shift; - cfg.write_masked(i, (val << shift) as u32, mask as u32); + let (at, reg) = Self::cfg_target(off, size); + if !self.bus.config_write(at, reg as usize, size, val as u32) { + self.config_master_timeout(at.device, reg); + } true } // ---- device windows ---- - /// One PCI cycle of `host.len()` bytes (at most the bus width) at window - /// offset `woff`. `host` is in CPU address order, most significant byte - /// first. With the programmed swap set, host byte k is PCI byte k; clear, - /// a 32-bit device sees it at k ^ 3 within its word and a 64-bit one at - /// 7 - k within its doubleword (the specification's big-endian tables 65 - /// to 80), and an access narrower than the device moves to - /// offset ^ (width - size). - #[allow(clippy::too_many_arguments)] - fn pio_cycle(&self, slot: usize, space: Space, base: u32, woff: u32, width: usize, swap: bool, host: &mut [u8], write: bool) -> bool { - let n = host.len(); - let target = base.wrapping_add(if swap { woff } else { woff ^ (width - n) as u32 }); - let claimed = self.slots.read().iter().flatten().find_map(|d| { - let (bar, boff) = d.config().decode(space, target)?; - Some((d.clone(), bar, boff)) - }); - let Some((dev, bar, boff)) = claimed else { - // Nobody drove device select: a master time-out. - self.irq_locked(|st, p| self.pci_error(st, p, INT_PCI_MASTER_TIMEOUT, slot as u32, target as u64)); - return false; - }; - Self::bar_cycle(&*dev, bar, boff, swap, host, write); - true - } - - /// The cycle `pio_cycle` found a target for: `host` (CPU order) to or - /// from BAR `bar` of `dev` at `boff`, lanes as `swap` says. - fn bar_cycle(dev: &dyn PciBusDevice, bar: usize, boff: u32, swap: bool, host: &mut [u8], write: bool) { - let n = host.len(); - // PCI bytes in address order; a function sees little-endian values. - let mut pci = [0u8; 8]; - if write { - for i in 0..n { - pci[i] = host[if swap { i } else { n - 1 - i }]; - } - } - let mut done = 0; - while done < n { - let chunk = (n - done).min(4); - let at = boff + done as u32; - if write { - let v = pci[done..done + chunk].iter().rev().fold(0u32, |a, &b| a << 8 | b as u32); - dev.bar_write(bar, at, v, chunk); - } else { - let v = dev.bar_read(bar, at, chunk); - for i in 0..chunk { - pci[done + i] = (v >> (8 * i)) as u8; - } - } - done += chunk; - } - if !write { - for i in 0..n { - host[i] = pci[if swap { i } else { n - 1 - i }]; - } - } - } - /// A processor access of `size` bytes to slot `slot`'s window at window - /// offset `woff`; `val` is `Some` for a write. A doubleword to a 32-bit - /// device is two word cycles, lower address first, carrying host bytes 0 - /// to 3 and then 4 to 7. A read that fails on its first cycle ends - /// there; a write posts, so both cycles go out and it never fails. + /// offset `woff`; `val` is `Some` for a write. The bus finds the BAR the + /// address reaches and the window's byte order carries the bytes. Where + /// nothing drives device select the cycle ends in a master time-out: a read + /// fails, and a write posts and never does. A doubleword to a 32-bit device + /// is two word cycles, and a read that fails on its first ends there while + /// a write sends both. fn window_access(&self, slot: usize, woff: u32, size: usize, val: Option) -> Option { let dev_reg = self.st.lock().r(reg::device(slot as u32)); let base = (dev_reg & DEV_WINDOW_BASE_MASK) << 20; - let width = if dev_reg & DEV_64BIT_DEVICE != 0 { 8 } else { 4 }; - let swap = dev_reg & DEV_SWAP_PROGRAMMED != 0; let space = if dev_reg & DEV_MEMORY_SPACE != 0 { Space::Mem } else { Space::Io }; - Self::cycles(width, woff, size, val, |woff, host, write| { - self.pio_cycle(slot, space, base, woff, width, swap, host, write) - }) - } - - /// A processor access of `size` bytes at window offset `woff` as the - /// PCI cycles a `width`-byte device takes, each made by `cycle(offset, - /// host bytes, write)`. - fn cycles(width: usize, woff: u32, size: usize, val: Option, mut cycle: impl FnMut(u32, &mut [u8], bool) -> bool) -> Option { - let write = val.is_some(); - let mut host = [0u8; 8]; - if let Some(v) = val { - for (k, b) in host.iter_mut().enumerate().take(size) { - *b = (v >> (8 * (size - 1 - k))) as u8; - } + let order = Self::order_of(dev_reg); + if let Some((f, bar, off)) = self.bus.decode(space, base.wrapping_add(woff)) { + return Some(match val { + None => order.read(f, bar, off, size), + Some(v) => { + order.write(f, bar, off, size, v); + 0 + } + }); } - let ok = if size > width { - let (a, b) = host.split_at_mut(4); - let first = cycle(woff, a, write); - if first || write { - let second = cycle(woff + 4, &mut b[..4], write); - first && second - } else { - false + // The cycle's PCI address is where the byte order sends it: a byte stream as + // given, word values with the access moved to the device's own byte position. + let width = if dev_reg & DEV_64BIT_DEVICE != 0 { 8 } else { 4 }; + let aim = |woff: u32, n: usize| base.wrapping_add(if order == ByteOrder::Stream { woff } else { woff ^ (width - n) as u32 }); + let miss = |target: u32| self.irq_locked(|st, p| self.pci_error(st, p, INT_PCI_MASTER_TIMEOUT, slot as u32, target as u64)); + if size > width { + miss(aim(woff, 4)); + if val.is_some() { + miss(aim(woff + 4, 4)); } } else { - cycle(woff, &mut host[..size], write) - }; - if write { - return Some(0); + miss(aim(woff, size)); + } + val.map(|_| 0) + } + + /// A processor access to PCI memory address `pci` through the 32-bit + /// alias. As through a window, nothing driving device select is a + /// master time-out: a read fails and a write posts. + fn alias_access(&self, pci: u32, size: usize, val: Option) -> Option { + let order = Self::alias_order(self.st.lock().r(reg::CONTROL)); + if let Some((f, bar, off)) = self.bus.decode(Space::Mem, pci) { + return Some(match val { + None => order.read(f, bar, off, size), + Some(v) => { + order.write(f, bar, off, size, v); + 0 + } + }); } - ok.then(|| host[..size].iter().fold(0u64, |a, &b| a << 8 | b as u64)) + let aim = |a: u32, n: usize| if order == ByteOrder::Stream { a } else { a ^ (4 - n.min(4)) as u32 }; + self.config_error(0, aim(pci, size) as u64); + val.map(|_| 0) } // ---- flash ---- @@ -1171,7 +1194,7 @@ impl Bridge { } /// The port is 16 bits wide, so a read of any size is the part's words - /// read one cycle at a time with the requested byte lanes kept. + /// read one cycle at a time with the requested byte order kept. fn flash_read(&self, woff: u32, size: usize) -> Option { let end = woff + size as u32; let mut result = 0u64; @@ -1314,6 +1337,9 @@ impl Bridge { }, }; if !good { + crate::dlog!(LogModule::Bridge, "bridge: slot {slot} {} of {step} bytes at PCI {a:#x} reached no memory ({})", + if matches!(buf, DmaBuf::Read(_)) { "read" } else { "write" }, + match target { None => "no translation".to_string(), Some((x, _, w)) => format!("widget {w:#x} address {x:#x}") }); ok = false; if let DmaBuf::Read(b) = &mut buf { b[done..done + step].fill(0); @@ -1411,7 +1437,12 @@ impl Bridge { /// is a bus error. fn read(&self, off: u32, size: usize) -> Option { match off { - 0..=0xffff => self.reg_read(off, size), + 0..=0xffff => { + let v = self.reg_read(off, size); + crate::dlog!(LogModule::Bridge, "bridge: read {} /{size} -> {}", reg_name(off & !7), + v.map_or("bus error".to_string(), |v| format!("{v:#x}"))); + v + } ATE_BASE..=0x10fff => self.ate_read(off - ATE_BASE, size), ATE_LOW_BASE..=0x11fff => self.ate_low_read(off - ATE_LOW_BASE, size), _ if (CFG0_BASE..CFG0_BASE + CFG0_SIZE).contains(&off) => self.cfg_read(off - CFG0_BASE, size), @@ -1422,6 +1453,8 @@ impl Bridge { let slot = window_slot(off)?; self.window_access(slot as usize, off - device_window_offset(slot), size, None) } + // Behind the device windows: probe traffic, not the PIO path. + _ if (TYPE1_BASE..TYPE1_BASE + TYPE1_SIZE).contains(&off) => self.type1_read(off - TYPE1_BASE, size), FLASH_BASE..=0xff_ffff => self.flash_read(off - FLASH_BASE, size), _ => None, } @@ -1429,21 +1462,32 @@ impl Bridge { /// As [`Bridge::read`]; the result is a bus status. fn write(&self, off: u32, size: usize, val: u64) -> u32 { + let swap_was = (off & !7 == reg::CONTROL).then(|| self.st.lock().r(reg::CONTROL) & CONTROL_MEM_SWAP); let ok = match off { - 0..=0xffff => self.reg_write(off, size, val), - ATE_BASE..=0x10fff => self.ate_write(off - ATE_BASE, size, val), + 0..=0xffff => { + crate::dlog!(LogModule::Bridge, "bridge: write {} /{size} {val:#x}", reg_name(off & !7)); + self.reg_write(off, size, val) + } + ATE_BASE..=0x10fff => { + crate::dlog!(LogModule::Bridge, "bridge: write translation entry {} /{size} {val:#x}", (off - ATE_BASE) / 8); + self.ate_write(off - ATE_BASE, size, val) + } _ if (CFG0_BASE..CFG0_BASE + CFG0_SIZE).contains(&off) => self.cfg_write(off - CFG0_BASE, size, val), _ if (SSRAM_BASE..SSRAM_BASE + SSRAM_SIZE).contains(&off) => true, _ if (DEVICE_WINDOW_BASE..FLASH_BASE).contains(&off) => match window_slot(off) { Some(s) => self.window_access(s as usize, off - device_window_offset(s), size, Some(val)).is_some(), None => false, }, + _ if (TYPE1_BASE..TYPE1_BASE + TYPE1_SIZE).contains(&off) => self.type1_write(off - TYPE1_BASE, size, val), FLASH_BASE..=0xff_ffff => self.flash_write(off - FLASH_BASE, size, val), _ => false, }; - let device_reg = (reg::device(0)..reg::device(DEVICES as u32)).contains(&(off & !7)); - if device_reg || (CFG0_BASE..CFG0_BASE + CFG0_SIZE).contains(&off) { - self.publish_bars(); + // A Device(x) write moves or re-aims a window, and a change of + // Control's memory swap bit re-aims the alias. Configuration writes + // update the mappings inside the bus. + let swap_changed = swap_was.is_some_and(|was| self.st.lock().r(reg::CONTROL) & CONTROL_MEM_SWAP != was); + if (reg::device(0)..reg::device(DEVICES as u32)).contains(&(off & !7)) || swap_changed { + self.bus.set_host_windows(&|| self.windows()); } if ok { BUS_OK @@ -1463,11 +1507,36 @@ impl Bridge { } } -/// A pin of the chip, 0..8, driven as a level (see [`Bridge::pin_line`], -/// which holds the Bridge weakly). -impl InterruptController for Bridge { - fn set_line(&self, line: u32, level: bool) { - self.set_wire_irq(line, level); +/// What the bus asks of its host: the DMA of a requester (the swap bits that +/// steer a transfer are its own slot's Device(x) register), and a pin's +/// level, as the OR of everything routed to it. The bus calls it with every +/// delivery to one pin serialised and in order, outside any lock of ours. +impl PciHost for Bridge { + fn dma_for(&self, requester: u8) -> Arc { + Arc::new(SlotDma { bridge: self.me.clone(), slot: requester as usize }) + } + + fn set_irq(&self, pin: u8, level: bool) { + self.irq_locked(|st, p| self.update_pin(st, p, pin as usize, level)); + } +} + +impl crate::traits::Device for Bridge { + fn step(&self, _cycles: u64) {} + fn stop(&self) {} + fn start(&self) {} + fn is_running(&self) -> bool { true } + fn get_clock(&self) -> u64 { 0 } + + fn register_commands(&self) -> Vec<(String, String)> { + vec![("pci".to_string(), "Bridge PCI bus: pci status | pci config ".to_string())] + } + + fn execute_command(&self, cmd: &str, args: &[&str], mut w: Box) -> Result<(), String> { + if cmd != "pci" { + return Err("Command not found".to_string()); + } + self.bus.command(args, &mut w) } } @@ -1482,40 +1551,78 @@ impl crate::traits::Resettable for Bridge { /// The widget window is decoded from the full physical address. const WINDOW_MASK: u32 = WIDGET_SMALL_SIZE - 1; -/// A device window's lanes, as its Device(x) register sets them: the -/// device's width and the programmed swap. What [`PciBar`] entries drive -/// a function through, the same cycles `window_access` makes once it has -/// found the target. -struct Lanes { - width: usize, - swap: bool, -} +/// The widget's 2 GB window: the small window's 16 MB, then nothing up to +/// the 32-bit alias of PCI memory. Runs a BAR decodes whole are mapped over +/// it; this answers the rest. +struct LongWindow(Arc); + +impl LongWindow { + fn read(&self, addr: u64, size: usize) -> Option { + match (addr & LONG_MASK) as u32 { + off @ 0..WIDGET_SMALL_SIZE => self.0.read(off, size), + off @ PIO32_ALIAS.. => self.0.alias_access(off - PIO32_ALIAS, size, None), + // Approximation: the specification calls 16 MB to 1 GB reserved + // (p. 52) and does not say what an access there does; a bus + // error here. + _ => None, + } + } -impl Lanes { - fn access(&self, f: &dyn PciBusDevice, bar: usize, off: u32, size: usize, val: Option) -> u64 { - let (width, swap) = (self.width, self.swap); - Bridge::cycles(width, off, size, val, |o, host, write| { - let boff = if swap { o } else { o ^ (width - host.len()) as u32 }; - Bridge::bar_cycle(f, bar, boff, swap, host, write); - true - }) - .unwrap_or(0) + fn write(&self, addr: u64, size: usize, val: u64) -> u32 { + match (addr & LONG_MASK) as u32 { + off @ 0..WIDGET_SMALL_SIZE => self.0.write(off, size, val), + off @ PIO32_ALIAS.. => { + self.0.alias_access(off - PIO32_ALIAS, size, Some(val)); + BUS_OK + } + _ => BUS_ERR, + } } } -impl PciLanes for Lanes { - fn read(&self, f: &dyn PciBusDevice, bar: usize, off: u32, size: usize) -> u64 { - self.access(f, bar, off, size, None) +impl BusDevice for LongWindow { + /// A configuration master time-out reached through the window's first + /// 16 MB is explained as through the small window. + fn describe_fault(&self, addr: u64, size: usize) -> Option { + let off = addr & LONG_MASK; + if off < WIDGET_SMALL_SIZE as u64 { self.0.describe_fault(off, size) } else { None } } - fn write(&self, f: &dyn PciBusDevice, bar: usize, off: u32, size: usize, val: u64) { - self.access(f, bar, off, size, Some(val)); + fn read8(&self, addr: u64) -> BusRead8 { + self.read(addr, 1).map_or(BusRead8::err(), |v| BusRead8::ok(v as u8)) + } + fn read16(&self, addr: u64) -> BusRead16 { + self.read(addr, 2).map_or(BusRead16::err(), |v| BusRead16::ok(v as u16)) + } + fn read32(&self, addr: u64) -> BusRead32 { + self.read(addr, 4).map_or(BusRead32::err(), |v| BusRead32::ok(v as u32)) + } + fn read64(&self, addr: u64) -> BusRead64 { + self.read(addr, 8).map_or(BusRead64::err(), BusRead64::ok) + } + fn write8(&self, addr: u64, val: u8) -> u32 { + self.write(addr, 1, val as u64) + } + fn write16(&self, addr: u64, val: u16) -> u32 { + self.write(addr, 2, val as u64) + } + fn write32(&self, addr: u64, val: u32) -> u32 { + self.write(addr, 4, val as u64) + } + fn write64(&self, addr: u64, val: u64) -> u32 { + self.write(addr, 8, val) } } impl BusDevice for Bridge { fn describe_fault(&self, addr: u64, size: usize) -> Option { let off = addr as u32 & WINDOW_MASK; - if (CFG0_BASE..CFG0_BASE + CFG0_SIZE).contains(&off) { self.cfg_fault(off - CFG0_BASE, size) } else { None } + if (CFG0_BASE..CFG0_BASE + CFG0_SIZE).contains(&off) { + self.cfg_fault(off - CFG0_BASE, size) + } else if (TYPE1_BASE..TYPE1_BASE + TYPE1_SIZE).contains(&off) { + self.type1_fault(off - TYPE1_BASE, size) + } else { + None + } } fn read8(&self, addr: u64) -> BusRead8 { @@ -1597,29 +1704,24 @@ impl BusDevice for RawFlash { } // The Bridge's state for a snapshot: its registers, address translation -// entries, interrupt pins and latched errors, its MicroLAN bus, and the -// configuration header of every function on its bus (reached through the -// Bridge, and not in the functions' own saved state). Loading puts the -// headers back first and then publishes the BAR pages from them, so the -// native PIO decode is derived, not stored. The flash parts behind its -// chip selects save themselves. +// entries, interrupt pin levels and latched errors, its MicroLAN bus, and +// the PCI bus (the configuration header of every function on it and the +// levels of the pins and board IRQs into the Bridge). Loading puts the headers +// back first and then publishes the BAR pages from them, so the native PIO +// decode is derived, not stored. The flash parts behind its chip selects +// save themselves. impl Saveable for Bridge { fn save_state(&self) -> toml::Value { let microlan = self.microlan.lock().as_ref().map_or(toml::Value::Boolean(false), |bus| bus.save_state()); - let configs: Vec = self.slots.read().iter() - .map(|f| f.as_ref().map_or(toml::Value::Boolean(false), |f| f.config().save_state())) - .collect(); let st = self.st.lock(); - let lines: Vec = st.pci_lines.iter().map(|&l| u8::from(l)).collect(); StateTable::new() .u32s("reg", &st.reg) .u64s("ate", &st.ate) .u32("int_error", st.int_error) - .u8s("pins", &[st.pin_level, st.pci_pin_level, st.gpio_pin_level, st.dma_64bit]) - .u8s("pci_lines", &lines) + .u8s("pins", &[st.pin_level, st.dma_64bit]) .u64("dma_pci_address", st.dma_pci_address) .u32("dma_device", st.dma_device) - .value("functions", toml::Value::Array(configs)) + .value("bus", self.bus.save_state()) .value("microlan", microlan) .u32("revision", self.revision.load(Ordering::Acquire)) .build() @@ -1634,19 +1736,7 @@ impl Saveable for Bridge { if ate.len() != ATE_ENTRIES { return Err(format!("ate: {} entries, expected {ATE_ENTRIES}", ate.len())); } - let lines = field_u8s::<{ DEVICES * 4 }>(v, "pci_lines")?; - let configs = field(v, "functions")?.as_array().ok_or("functions: not a list")?; - let slots = self.slots.read().clone(); - if configs.len() != slots.len() { - return Err(format!("functions: {} slots, expected {}", configs.len(), slots.len())); - } - for (slot, (f, saved)) in slots.iter().zip(configs).enumerate() { - match (f, saved.as_bool()) { - (Some(f), None) => f.config().load_state(saved).map_err(|e| format!("slot {slot}: {e}"))?, - (None, Some(false)) => {} - _ => return Err(format!("slot {slot}: the saved bus has a different function there")), - } - } + self.bus.load_state(field(v, "bus")?)?; let saved = field(v, "microlan")?; match (self.microlan.lock().as_mut(), saved.as_bool()) { (Some(bus), None) => bus.load_state(saved).map_err(|e| format!("MicroLAN: {e}"))?, @@ -1654,20 +1744,17 @@ impl Saveable for Bridge { _ => return Err("the saved Bridge has a different MicroLAN bus".into()), } { - let [pin_level, pci_pin_level, gpio_pin_level, dma_64bit] = field_u8s::<4>(v, "pins")?; + let [pin_level, dma_64bit] = field_u8s::<2>(v, "pins")?; let mut st = self.st.lock(); st.reg = field_u32s(v, "reg")?; st.ate.copy_from_slice(&ate); st.int_error = field_u32(v, "int_error")?; st.pin_level = pin_level; - st.pci_pin_level = pci_pin_level; - st.gpio_pin_level = gpio_pin_level; st.dma_64bit = dma_64bit; - st.pci_lines = lines.map(|l| l != 0); st.dma_pci_address = field_u64(v, "dma_pci_address")?; st.dma_device = field_u32(v, "dma_device")?; } - self.publish_bars(); + self.bus.set_host_windows(&|| self.windows()); Ok(()) } } diff --git a/src/dev/bridge_tests.rs b/src/dev/bridge_tests.rs index 8bc2e7a21..a2cf2c974 100644 --- a/src/dev/bridge_tests.rs +++ b/src/dev/bridge_tests.rs @@ -9,7 +9,8 @@ use super::*; use crate::ip30::{HEART_HOST, WIDGET_HEART}; use crate::irq::IrqLine; -use crate::pci::{Bar, PciConfig, PCI_COMMAND_IO, PCI_COMMAND_MEMORY, PCI_INTA}; +use crate::pci::PciToPciBridge; +use crate::pci::{Bar, BoardIrq, DevFn, PciConfig, PciFunction, PciFunctionState, PCI_COMMAND_IO, PCI_COMMAND_MEMORY, PCI_INTA}; use crate::xtalk::{Upstream, XtalkHost}; /// HEART stand-in: the interrupt packets it received, in order. A line is @@ -99,10 +100,8 @@ const FIXED: u32 = 0x20; /// that stores little-endian bytes, a fixed word at 0x20, an interrupt line /// and the DMA it was given. struct Card { - cfg: PciConfig, + pci: PciFunctionState, regs: Mutex<[u8; 0x1000]>, - irq: Mutex>, - dma: Mutex>>, } impl Card { @@ -110,17 +109,15 @@ impl Card { let mut regs = [0u8; 0x1000]; regs[FIXED as usize..FIXED as usize + 4].copy_from_slice(&0x1234_5678u32.to_le_bytes()); Arc::new(Self { - cfg: PciConfig::new( + pci: PciFunctionState::new(PciConfig::new( CARD_VENDOR as u16, CARD_DEVICE as u16, 0x01_0000, 1, [Bar::Io(0x100), Bar::Mem(0x1000), Bar::Unused, Bar::Unused, Bar::Unused, Bar::Unused], pin, - ), + )), regs: Mutex::new(regs), - irq: Mutex::new(None), - dma: Mutex::new(None), }) } @@ -129,17 +126,17 @@ impl Card { } fn raise(&self, level: bool) { - self.irq.lock().as_ref().unwrap().set(level); + self.pci.irq().unwrap().set(level); } fn dma(&self) -> Arc { - self.dma.lock().clone().unwrap() + self.pci.dma().unwrap().clone() } } -impl PciBusDevice for Card { - fn config(&self) -> &PciConfig { - &self.cfg +impl PciFunction for Card { + fn state(&self) -> &PciFunctionState { + &self.pci } fn bar_read(&self, _bar: usize, off: u32, width: usize) -> u32 { @@ -153,13 +150,6 @@ impl PciBusDevice for Card { self.regs.lock()[off as usize..off as usize + width].copy_from_slice(&val.to_le_bytes()[..width]); } - fn connect_interrupt(&self, line: IrqLine) { - *self.irq.lock() = Some(line); - } - - fn connect_dma(&self, dma: Arc) { - *self.dma.lock() = Some(dma); - } } struct Rig { @@ -176,7 +166,7 @@ impl Rig { fn with_host(host: XtalkHost) -> Self { let sink = Arc::new(Sink::default()); let mem = Arc::new(Mem::new()); - Rig { bridge: Bridge::new(Upstream::single(host, mem.clone(), sink.clone())), sink, mem } + Rig { bridge: Bridge::new(Upstream::single(host, mem.clone(), sink.clone()), &crate::ip30::PCI0), sink, mem } } /// Write register `off` (a slot base such as `reg::CONTROL`). @@ -188,6 +178,11 @@ impl Rig { self.bridge.reset(); } + /// Fit `f` in `slot` on the Bridge's bus. + fn fit(&self, slot: u8, f: Arc) { + self.bridge.bus().attach(DevFn { device: slot, function: 0 }, f); + } + fn get(&self, off: u32) -> u32 { let r = self.bridge.read32(u64::from(off + 4)); assert!(r.is_ok()); @@ -227,7 +222,7 @@ impl Rig { /// at 0x2000 and both enabled, and the slot's window aimed at memory. fn card(&self, slot: u8, pin: u8) -> Arc { let c = Card::new(pin); - self.bridge.attach(slot, c.clone()); + self.fit(slot, c.clone()); self.cfg_write(slot as u32, 0x10, 4, 0x2000); self.cfg_write(slot as u32, 0x14, 4, 0x1300_0000); self.cfg_write(slot as u32, 0x04, 2, PCI_COMMAND_IO | PCI_COMMAND_MEMORY); @@ -410,7 +405,7 @@ fn the_one_wire_master_runs_a_bit_slot_per_write() { #[test] fn configuration_space() { let r = Rig::new(); - r.bridge.attach(1, Card::new(PCI_INTA)); + r.fit(1, Card::new(PCI_INTA)); assert_eq!(r.cfg_read(1, 0, 2), CARD_VENDOR); assert_eq!(r.cfg_read(1, 2, 2), CARD_DEVICE); assert_eq!(r.cfg_read(1, 0, 4), CARD_VENDOR | (CARD_DEVICE << 16)); @@ -423,7 +418,7 @@ fn configuration_space() { fn narrow_configuration_writes_merge_into_the_word() { let r = Rig::new(); let c = Card::new(PCI_INTA); - r.bridge.attach(1, c.clone()); + r.fit(1, c.clone()); r.cfg_write(1, 0x3c, 4, 0x0000_0100); r.cfg_write(1, 0x3c, 1, 0x0b); assert_eq!(r.cfg_read(1, 0x3c, 4), 0x0000_010b, "pin kept, line written"); @@ -453,7 +448,7 @@ fn an_empty_slot_fails_the_read_and_posts_the_write() { r.set(reg::INT_RESET, INT_RESET_PCI_GRP | INT_RESET_MULTI_ERR); assert_eq!(r.get(reg::INT_STATUS), 0); // A function other than 0 is not there either. - r.bridge.attach(2, Card::new(PCI_INTA)); + r.fit(2, Card::new(PCI_INTA)); assert!(!r.bridge.read32(u64::from(Rig::cfg(2, 0x100))).is_ok()); } @@ -573,6 +568,198 @@ fn an_empty_slot_config_read_names_the_master_timeout() { assert_eq!(r.bridge.describe_fault(0x1f00_0000 + at(3, 0), 4), empty(3)); } +// ---- type 1 configuration ---- + +impl Rig { + /// A PCI-to-PCI bridge in `slot` with a card at device 0 behind it. + fn bridge_with_card_behind(&self, slot: u8) -> Arc { + let p2p = PciToPciBridge::new(self.bridge.bus(), DevFn { device: slot, function: 0 }, 0x1011, 0x0026); + let c = Card::new(PCI_INTA); + p2p.secondary().attach(DevFn { device: 0, function: 0 }, c.clone()); + self.fit(slot, p2p); + c + } + + /// Aim the type 1 aperture at `bus` and `device`. + fn type1_target(&self, bus: u32, device: u32) { + self.set(reg::PCI_TYPE1_CONFIG, bus << 16 | device << 11); + assert_eq!(self.get(reg::PCI_TYPE1_CONFIG), bus << 16 | device << 11); + } + + fn type1_read(&self, func: u32, reg: u32, size: u32) -> Option { + let a = u64::from(TYPE1_BASE + (func << 8) + Self::aim(reg, size)); + match size { + 1 => Some(self.bridge.read8(a)).filter(|r| r.is_ok()).map(|r| r.data as u32), + 2 => Some(self.bridge.read16(a)).filter(|r| r.is_ok()).map(|r| r.data as u32), + _ => Some(self.bridge.read32(a)).filter(|r| r.is_ok()).map(|r| r.data), + } + } + + fn type1_write(&self, func: u32, reg: u32, size: u32, val: u32) { + let a = u64::from(TYPE1_BASE + (func << 8) + Self::aim(reg, size)); + let s = match size { + 1 => self.bridge.write8(a, val as u8), + 2 => self.bridge.write16(a, val as u16), + _ => self.bridge.write32(a, val), + }; + assert_eq!(s, BUS_OK); + } +} + +/// IRIX 6.5.30 on the Octane finds a bridge in slot 5, numbers its buses +/// with byte writes (primary 0, secondary and subordinate 0x10) and scans +/// all 32 devices of bus 0x10 through the type 1 aperture. The card behind +/// the bridge answers at device 0 and nothing else does: the bridge returns +/// all ones for the rest. +#[test] +fn the_type_1_aperture_reaches_the_bus_behind_a_bridge() { + let r = Rig::new(); + let c = r.bridge_with_card_behind(5); + assert_eq!(r.cfg_read(5, 0x08, 4) >> 8, 0x06_0400, "a PCI-to-PCI bridge"); + assert_eq!(r.cfg_read(5, 0x0e, 1) & 0x7f, 1, "type 1 header"); + r.cfg_write(5, 0x18, 1, 0x00); + r.cfg_write(5, 0x19, 1, 0x10); + r.cfg_write(5, 0x1a, 1, 0x10); + + let found: Vec = (0..32) + .filter(|&d| { + r.type1_target(0x10, d); + r.type1_read(0, 0, 4) != Some(0xffff_ffff) + }) + .collect(); + assert_eq!(found, [0]); + + r.type1_target(0x10, 0); + assert_eq!(r.type1_read(0, 0, 4), Some(CARD_VENDOR | CARD_DEVICE << 16)); + assert_eq!(r.type1_read(0, 2, 2), Some(CARD_DEVICE), "the same byte order as type 0"); + assert_eq!(r.type1_read(0, 1, 1), Some(CARD_VENDOR >> 8)); + assert_eq!(r.type1_read(1, 0, 4), Some(0xffff_ffff), "the card has no function 1"); + + // Size and place the card's memory BAR through the aperture. + r.type1_write(0, 0x14, 4, 0xffff_ffff); + assert_eq!(r.type1_read(0, 0x14, 4), Some(0xffff_f000)); + r.type1_write(0, 0x14, 4, 0x1300_0000); + r.type1_write(0, 0x04, 2, PCI_COMMAND_MEMORY); + assert_eq!(c.state().config().bar_base(1), 0x1300_0000); + assert_eq!(r.type1_read(0, 0x04, 2), Some(PCI_COMMAND_MEMORY)); + + // Open the bridge's memory window over it: slot 5's device window + // reaches the card's registers. + r.cfg_write(5, 0x20, 4, 0x1300_1300); + r.cfg_write(5, 0x04, 2, PCI_COMMAND_MEMORY); + r.set(reg::device(5), DEV_MEMORY_SPACE | 0x130); + assert_eq!(r.bridge.read32(u64::from(Rig::win(5) + FIXED)).data, 0x1234_5678); +} + +/// A type 1 cycle no bridge claims ends in a master time-out, like an empty +/// slot: bus 0 itself (type 0 functions ignore type 1 cycles) and a bus +/// number outside every bridge's range. +#[test] +fn a_type_1_cycle_nobody_claims_is_a_master_timeout() { + let r = Rig::new(); + r.bridge_with_card_behind(5); + r.cfg_write(5, 0x18, 4, 0x0001_0100); + r.set(reg::INT_ENABLE, INT_PCI_MASTER_TIMEOUT); + + r.type1_target(0, 5); + assert_eq!(r.type1_read(0, 0, 4), None, "the bridge itself is on bus 0"); + assert_eq!(r.bridge.describe_fault(u64::from(TYPE1_BASE), 4), + Some("PCI type 1 config master timeout: no bridge reaches bus 0".to_string())); + r.type1_target(2, 0); + r.type1_write(0, 0x04, 2, 0); + assert_eq!(r.type1_read(0, 0, 4), None); + assert_ne!(r.get(reg::INT_STATUS) & INT_PCI_MASTER_TIMEOUT, 0); + // The capture holds the first: a type 1 address (bus 0, device 5). + assert_eq!(r.get(reg::PCI_ERR_LOWER), 5 << 11 | 1); + + r.type1_target(1, 0); + assert!(r.type1_read(0, 0, 4).is_some()); + assert_eq!(r.bridge.describe_fault(u64::from(TYPE1_BASE), 4), None); +} + +/// Behind a PCI-to-PCI bridge an unclaimed type 1 cycle ends in master abort +/// at the bridge, not at the Bridge: a read returns all ones, a write is +/// dropped, the bridge's secondary status records the abort, and no master +/// time-out is raised (PCI-to-PCI Bridge Architecture, master abort handling +/// with Master-Abort Mode clear). +#[test] +fn an_absent_device_behind_a_pci_bridge_master_aborts_at_the_bridge() { + let r = Rig::new(); + r.bridge_with_card_behind(5); + r.cfg_write(5, 0x18, 4, 0x0001_0100); + r.set(reg::INT_ENABLE, INT_PCI_MASTER_TIMEOUT); + assert_eq!(r.cfg_read(5, 0x1c, 4) >> 29 & 1, 0, "no abort yet"); + + r.type1_target(1, 9); + assert_eq!(r.type1_read(0, 0, 4), Some(0xffff_ffff)); + assert_eq!(r.type1_read(0, 0, 2), Some(0xffff)); + assert_eq!(r.type1_read(0, 3, 1), Some(0xff)); + r.type1_write(0, 0x04, 2, PCI_COMMAND_MEMORY); + assert_eq!(r.bridge.describe_fault(u64::from(TYPE1_BASE), 4), None); + assert_eq!(r.get(reg::INT_STATUS) & INT_PCI_MASTER_TIMEOUT, 0, "the Bridge saw every cycle complete"); + assert_eq!(r.cfg_read(5, 0x1c, 4) >> 29 & 1, 1, "received master abort in secondary status"); + + // Write one to clear. + r.cfg_write(5, 0x1c, 4, 1 << 29); + assert_eq!(r.cfg_read(5, 0x1c, 4) >> 29 & 1, 0); + + // The Bridge's own bus is unchanged: nothing there is a master time-out. + r.type1_target(0, 9); + assert_eq!(r.type1_read(0, 0, 4), None); + assert_ne!(r.get(reg::INT_STATUS) & INT_PCI_MASTER_TIMEOUT, 0); +} + +// ---- the 2 GB window ---- + +/// Through the widget's 2 GB window the first 16 MB is the small window +/// again, and offset 0x4000_0000 on is PCI memory from address 0, with +/// the bytes placed by Control's memory swap bit (Bridge specification +/// pp. 52 and 55). IRIX 6.5.30 reaches a BAR this way when a bridge +/// window is too wide for the slot's device window. +#[test] +fn the_2gb_window_reaches_pci_memory_through_the_32_bit_alias() { + let r = Rig::new(); + let c = r.card(1, PCI_INTA); // memory BAR at PCI 0x1300_0000 + let long = r.bridge.long_window(); + let alias = |pci: u32| u64::from(PIO32_ALIAS + pci) + 0xf_8000_0000; + assert_eq!(long.read32(0xf_8000_0000 + u64::from(reg::IDENTIFICATION) + 4).data, r.get(reg::IDENTIFICATION)); + assert_eq!(long.read32(alias(0x1300_0000 + FIXED)).data, 0x1234_5678, "swap clear: word values"); + assert_eq!(long.write32(alias(0x1300_0000 + SCRATCH), 0x8b7a_6950), BUS_OK); + assert_eq!(c.peek32(SCRATCH as usize), 0x8b7a_6950); + assert_eq!(long.read8(alias(0x1300_0000 + (FIXED ^ 3))).data, 0x78, "a byte moves to its position"); + r.set(reg::CONTROL, r.get(reg::CONTROL) | CONTROL_MEM_SWAP); + assert_eq!(long.read32(alias(0x1300_0000 + FIXED)).data, 0x7856_3412, "swap set: a byte stream"); + // Between the small window and the alias, and PCI memory nothing decodes. + assert!(!long.read32(0xf_8000_0000 + 0x0100_0000).is_ok()); + r.set(reg::INT_ENABLE, INT_PCI_MASTER_TIMEOUT); + assert!(!long.read32(alias(0x2000_0000)).is_ok()); + assert_ne!(r.get(reg::INT_STATUS) & INT_PCI_MASTER_TIMEOUT, 0); + // Approximation: the capture names device 0, and the PCI address the + // cycle carried. + assert_eq!(r.get(reg::PCI_ERR_UPPER), 0); + assert_eq!(r.get(reg::PCI_ERR_LOWER), 0x2000_0000); +} + +/// A BAR the alias decodes whole is published as a run at its alias +/// offset, and the run answers as the 2 GB window does. +#[test] +fn a_bar_in_the_alias_is_a_run() { + let r = Rig::new(); + let runs = Arc::new(Mutex::new(Vec::new())); + let (sink, b) = (runs.clone(), Arc::downgrade(&r.bridge)); + r.bridge.bus().on_bar_mappings(Box::new(move |rs| { + let b = b.upgrade().unwrap(); + *sink.lock() = rs.iter().map(|x| (x.cpu..x.cpu + x.len, b.bar_device(x))).collect::>(); + })); + r.card(1, PCI_INTA); + let at = u64::from(PIO32_ALIAS + 0x1300_0000); + let run = runs.lock().iter().find(|x| x.0.contains(&at)).cloned().expect("the alias carries a run"); + assert_eq!(run.0, at..at + 0x1000); + let pa = 0xf_8000_0000 + at + u64::from(FIXED); + assert_eq!(run.1.read32(pa).data, r.bridge.long_window().read32(pa).data); + assert_eq!(run.1.read32(pa).data, 0x1234_5678); +} + #[test] fn a_disabled_command_register_hides_the_bars() { let r = Rig::new(); @@ -821,13 +1008,225 @@ fn pins_follow_the_slot_wiring_and_share_by_or() { fn a_directly_wired_input_shares_the_pins() { let r = Rig::new(); r.route(4, 33); - r.bridge.set_wire_irq(4, true); + r.bridge.bus().board_irq(BoardIrq(4)).set(true); assert_eq!(r.sink.take(), [(33, true)]); assert_eq!(r.get(reg::INT_STATUS) & 0xff, 1 << 4); - r.bridge.set_wire_irq(4, false); + r.bridge.bus().board_irq(BoardIrq(4)).set(false); assert_eq!(r.get(reg::INT_STATUS) & 0xff, 0); } +// ---- the PCI reset pins, Control 27:24 ---- + +/// A recording function at `slot`, command register enabled, and the +/// log of the RST# deliveries it has seen. +fn watcher(r: &Rig, slot: u8) -> (Arc>>, DevFn) { + use crate::pci::testing::TestCard; + let log = Arc::new(std::sync::Mutex::new(Vec::new())); + let at = DevFn { device: slot, function: 0 }; + let bars = [Bar::Mem(0x1000), Bar::Unused, Bar::Unused, Bar::Unused, Bar::Unused, Bar::Unused]; + let name = ["w0", "w1", "w2", "w3", "w4", "w5", "w6", "w7"][slot as usize]; + r.fit(slot, TestCard::logged(name, bars, 0, &log)); + r.bridge.bus().config_write(at, 4, 2, PCI_COMMAND_MEMORY); + (log, at) +} + +const RST_PINS: u32 = 0xf << 24; + +/// Control resets with every pin deasserted (1); writing it back with them +/// still deasserted resets nothing, so the PROM's first Control write +/// does not reset the functions. +#[test] +fn releasing_untouched_reset_pins_resets_nothing() { + let r = Rig::new(); + let (log, at) = watcher(&r, 0); + assert_eq!(r.get(reg::CONTROL) & RST_PINS, RST_PINS, "deasserted at reset"); + r.set(reg::CONTROL, r.get(reg::CONTROL)); + r.set(reg::CONTROL, r.get(reg::CONTROL) | DEV_ERROR_LOCK); + assert!(log.lock().unwrap().is_empty()); + assert_eq!(r.bridge.bus().config_read(at, 4, 2), Some(PCI_COMMAND_MEMORY), "and the header is as it was"); +} + +/// Pin `n` going from 1 to 0 resets the function at device `n`, once, and no +/// other: not another pin's device and not a device with no pin. Raising the +/// pin again resets nothing. (The pin-to-device wiring is inferred.) +#[test] +fn a_one_to_zero_on_reset_pin_n_resets_device_n_only_once() { + for pin in 0..4u8 { + let r = Rig::new(); + let watchers: Vec<_> = [0u8, 1, 2, 3, 6].iter().map(|&slot| (slot, watcher(&r, slot))).collect(); + let ctl = r.get(reg::CONTROL); + r.set(reg::CONTROL, ctl & !(1 << (24 + pin))); + for (slot, (log, at)) in &watchers { + if *slot == pin { + assert_eq!(*log.lock().unwrap(), [format!("w{slot} cmd=0x0")], "pin {pin}: header reset first, then the function"); + assert_eq!(r.bridge.bus().config_read(*at, 4, 2), Some(0)); + } else { + assert!(log.lock().unwrap().is_empty(), "pin {pin} left device {slot} alone"); + assert_eq!(r.bridge.bus().config_read(*at, 4, 2), Some(PCI_COMMAND_MEMORY)); + } + } + r.set(reg::CONTROL, ctl); + assert_eq!(watchers[pin as usize].1 .0.lock().unwrap().len(), 1, "pin {pin}: releasing it resets nothing"); + } +} + +/// On the Octane's own devices: pin 0 resets the first ISP1040 and leaves the +/// IOC3 (the console and Ethernet) and the other ISP alone. +#[test] +fn reset_pin_0_resets_device_0_and_leaves_the_ioc3_and_the_other_isp() { + use crate::pci::testing::TestCard; + let r = Rig::new(); + let bars = [Bar::Mem(0x1000), Bar::Unused, Bar::Unused, Bar::Unused, Bar::Unused, Bar::Unused]; + let log = Arc::new(std::sync::Mutex::new(Vec::new())); + // Devices 0 and 1 are the ISPs and 2 the IOC3 on the Octane. + for (slot, name) in [(0u8, "isp0"), (1, "isp1"), (2, "ioc3")] { + r.fit(slot, TestCard::logged(name, bars, 0, &log)); + r.bridge.bus().config_write(DevFn { device: slot, function: 0 }, 4, 2, PCI_COMMAND_MEMORY); + } + let ctl = r.get(reg::CONTROL); + r.set(reg::CONTROL, ctl & !(1 << 24)); + assert_eq!(*log.lock().unwrap(), ["isp0 cmd=0x0"]); + for slot in [1u8, 2] { + assert_eq!(r.bridge.bus().config_read(DevFn { device: slot, function: 0 }, 4, 2), Some(PCI_COMMAND_MEMORY)); + } +} + +#[test] +fn holding_a_pin_low_does_not_reset_again() { + let r = Rig::new(); + let (log1, _) = watcher(&r, 1); + let (log2, _) = watcher(&r, 2); + let ctl = r.get(reg::CONTROL); + r.set(reg::CONTROL, ctl & !(1 << 25)); + r.set(reg::CONTROL, ctl & !(1 << 25) | DEV_ERROR_LOCK); + r.set(reg::CONTROL, ctl & !(0x3 << 25)); + assert_eq!(log1.lock().unwrap().len(), 1, "holding pin 1 low is not a new 1 to 0"); + assert_eq!(log2.lock().unwrap().len(), 1, "pin 2 is"); +} + +/// The specification does not say that a Bridge core reset or a Crossbow +/// link reset pulses the pins, so neither resets the bus; the functions +/// keep their lines. +#[test] +fn a_core_reset_and_a_link_reset_run_no_bus_reset_and_keep_pin_levels() { + let r = Rig::new(); + let (log, at) = watcher(&r, 6); + let c = r.card(2, PCI_INTA); + r.route(2, 1); + c.raise(true); + r.reset(); + r.bridge.link_reset(); + assert!(log.lock().unwrap().is_empty()); + assert_eq!(r.bridge.bus().config_read(at, 4, 2), Some(PCI_COMMAND_MEMORY)); + assert_eq!(r.get(reg::INT_STATUS) & 0xff, 1 << 2, "the card still holds its line"); +} + +/// The wiring table is the rule the Bridge always had: pins A and C reach +/// the pin numbered as the slot, B and D that number with bit two flipped. +#[test] +fn octane_routes_equal_the_old_map_irq() { + let old = |slot: usize, pin0: usize| if pin0 & 1 != 0 { slot ^ 4 } else { slot }; + for slot in 0..DEVICES { + for pin in 1..=4usize { + let r = Rig::new(); + let c = r.card(slot as u8, pin as u8); + r.route(old(slot, pin - 1) as u32, 9); + c.raise(true); + assert_eq!(r.sink.take(), [(9, true)], "slot {slot} pin {pin}"); + assert_eq!(r.get(reg::INT_STATUS) & 0xff, 1 << old(slot, pin - 1)); + } + } +} + +/// A pin that falls and rises again while a packet is still on its way to +/// HEART reaches the Bridge as a fall and a rise, so the second raise is a +/// new interrupt with its own packet, not a level that never dropped. (The +/// guest may have cleared HEART's bit in between, and only a packet sets +/// it again.) +#[test] +fn pin_fall_then_rise_gives_the_bridge_one_heart_packet_each() { + use std::sync::mpsc::{channel, Receiver, Sender}; + /// HEART that holds its first packet until released. + struct Slow { + log: Mutex>, + stall: Mutex, Receiver<()>)>>, + } + impl InterruptController for Slow { + fn set_line(&self, line: u32, level: bool) { + self.log.lock().push((line, level)); + if let Some((begun, go)) = self.stall.lock().take() { + begun.send(()).unwrap(); + go.recv().unwrap(); + } + } + } + let heart = Arc::new(Slow { log: Mutex::new(Vec::new()), stall: Mutex::new(None) }); + let bridge = Bridge::new(Upstream::single(HEART_HOST, Arc::new(Mem::new()), heart.clone()), &crate::ip30::PCI0); + let r = Rig { bridge, sink: Arc::new(Sink::default()), mem: Arc::new(Mem::new()) }; + let c = r.card(2, PCI_INTA); + r.route(2, 1); + let (begun_tx, begun_rx) = channel(); + let (go_tx, go_rx) = channel(); + *heart.stall.lock() = Some((begun_tx, go_rx)); + let raiser = { + let c = c.clone(); + std::thread::spawn(move || c.raise(true)) + }; + begun_rx.recv().unwrap(); + c.raise(false); + c.raise(true); + go_tx.send(()).unwrap(); + raiser.join().unwrap(); + assert_eq!(*heart.log.lock(), [(1, true), (1, true)]); +} + +/// Slot 1 INTB and slot 5 INTA share pin 5: while the other line holds the +/// pin high, a fall and rise on one of them sends no packet. +#[test] +fn a_shared_pin_with_the_other_line_high_sends_no_packet() { + let r = Rig::new(); + let b1 = r.card(1, 2); + let a5 = r.card(5, PCI_INTA); + r.route(5, 40); + a5.raise(true); + assert_eq!(r.sink.take(), [(40, true)]); + b1.raise(true); + b1.raise(false); + b1.raise(true); + assert!(r.sink.take().is_empty(), "the pin never fell"); +} + +/// A Device(x) write is the next access's window and byte order: re-aimed to +/// another PCI base the old BAR is out of reach, and switching the +/// programmed swap changes the bytes the function sees. +#[test] +fn a_device_x_write_reaches_the_next_access() { + let r = Rig::new(); + let c = r.card(1, PCI_INTA); + let at = Rig::win(1) + SCRATCH; + assert_eq!(r.bridge.write32(u64::from(at), 0x0102_0304), BUS_OK); + assert_eq!(c.peek32(SCRATCH as usize), 0x0102_0304, "word values keep the value"); + r.set(reg::device(1), DEV_MEMORY_SPACE | DEV_SWAP_PROGRAMMED | 0x130); + assert_eq!(r.bridge.write32(u64::from(at), 0x0102_0304), BUS_OK); + assert_eq!(c.peek32(SCRATCH as usize), 0x0403_0201, "the programmed swap is a byte stream"); + r.set(reg::device(1), DEV_MEMORY_SPACE | 0x140); + assert!(!r.bridge.read32(u64::from(at)).is_ok(), "nothing decodes at the new base"); +} + +/// The IOC3's SuperIO interrupt is board wire 4, which shares the Bridge's +/// pin 4 with whatever PCI functions are wired to it. +#[test] +fn the_superio_wire_reaches_pin_4() { + let r = Rig::new(); + let pci = r.card(4, PCI_INTA); // slot 4 INTA is pin 4 too + r.route(4, 33); + r.bridge.bus().board_irq(BoardIrq(4)).set(true); + assert_eq!(r.sink.take(), [(33, true)]); + pci.raise(true); + r.bridge.bus().board_irq(BoardIrq(4)).set(false); + assert_eq!(r.get(reg::INT_STATUS) & 0xff, 1 << 4, "the card still holds it"); +} + /// A Bridge-only reset leaves the functions' lines alone: a card still /// holding INTA must show on the pin, and a later enable must send it. #[test] @@ -981,7 +1380,7 @@ fn the_programmed_widget_picks_the_host_behind_a_crossbow() { HostPort::new(HEART_HOST, mem_a.clone(), sink_a.clone()), HostPort::new(host_b, mem_b.clone(), sink_b.clone()), ]); - let r = Rig { bridge: Bridge::new(up), sink: sink_a.clone(), mem: mem_a }; + let r = Rig { bridge: Bridge::new(up, &crate::ip30::PCI0), sink: sink_a.clone(), mem: mem_a }; (r, mem_b, sink_b) }; let (r, mem_b, sink_b) = build(1 << 8 | 1 << 10 | 1 << 15); @@ -1195,21 +1594,16 @@ fn the_64_bit_direct_window_refuses_widget_zero_and_other_widgets() { // ---- lines into the Bridge, and the packets out of it ---- /// IOC3's SuperIO interrupt is wired to a pin directly; the line shares the -/// pin with PCI functions and holds the Bridge weakly. +/// pin with PCI functions and does not keep the Bridge alive. #[test] -fn a_pin_line_drives_its_pin_and_does_not_keep_the_bridge_alive() { +fn a_board_irq_line_drives_its_pin_and_does_not_keep_the_bridge_alive() { let r = Rig::new(); r.route(4, 33); - let line = r.bridge.pin_line(4); - assert_eq!(line.line(), 4); + let line = r.bridge.bus().board_irq(BoardIrq(4)); line.set(true); assert_eq!(r.sink.take(), [(33, true)]); assert_eq!(r.get(reg::INT_STATUS) & 0xff, 1 << 4); - // The Bridge is itself a controller whose line is the pin. - r.bridge.set_line(4, false); - assert_eq!(r.get(reg::INT_STATUS) & 0xff, 0); - // A line past the eight pins is dropped. - r.bridge.set_line(8, true); + line.set(false); assert_eq!(r.get(reg::INT_STATUS) & 0xff, 0); let weak = Arc::downgrade(&r.bridge); @@ -1231,11 +1625,11 @@ fn a_packet_reaches_heart_outside_the_state_lock() { } } let heart = Arc::new(Prying(Mutex::new(None), Mutex::new(Vec::new()))); - let bridge = Bridge::new(Upstream::single(HEART_HOST, Arc::new(Mem::new()), heart.clone())); + let bridge = Bridge::new(Upstream::single(HEART_HOST, Arc::new(Mem::new()), heart.clone()), &crate::ip30::PCI0); *heart.0.lock() = Some(bridge.clone()); let r = Rig { bridge, sink: Arc::new(Sink::default()), mem: Arc::new(Mem::new()) }; r.route(4, 3); - r.bridge.set_wire_irq(4, true); + r.bridge.bus().board_irq(BoardIrq(4)).set(true); assert_eq!(*heart.1.lock(), [1 << 4], "the status already shows the level"); // Errors leave the same way. r.set(reg::INT_ENABLE, 1 << 4 | INT_PCI_MASTER_TIMEOUT); @@ -1251,10 +1645,9 @@ fn edges_from_two_threads_reach_heart_in_decision_order() { let r = Rig::new(); r.route(4, 7); r.set(reg::INT_MODE, 1 << 4); - let b1 = r.bridge.clone(); - let b2 = r.bridge.clone(); - let t1 = std::thread::spawn(move || (0..2000).for_each(|_| { b1.set_wire_irq(4, true); b1.set_wire_irq(4, false); })); - let t2 = std::thread::spawn(move || (0..2000).for_each(|_| { b2.set_wire_irq(4, true); b2.set_wire_irq(4, false); })); + let (b1, b2) = (r.bridge.bus().board_irq(BoardIrq(4)), r.bridge.bus().board_irq(BoardIrq(4))); + let t1 = std::thread::spawn(move || (0..2000).for_each(|_| { b1.set(true); b1.set(false); })); + let t2 = std::thread::spawn(move || (0..2000).for_each(|_| { b2.set(true); b2.set(false); })); t1.join().unwrap(); t2.join().unwrap(); let packets = r.sink.take(); @@ -1274,7 +1667,7 @@ fn edges_from_two_threads_reach_heart_in_decision_order() { fn window_2_reaches_a_bar_the_way_the_prom_points_it() { let r = Rig::new(); let c = Card::new(PCI_INTA); - r.bridge.attach(2, c.clone()); + r.fit(2, c.clone()); // Nothing decodes before the setup. assert!(!r.bridge.read8(u64::from(Rig::win(2) + SCRATCH)).is_ok()); r.cfg_write(2, 0x04, 4, PCI_COMMAND_MEMORY); diff --git a/src/dev/crime_mem.rs b/src/dev/crime_mem.rs index 4b362456f..f8501c5cd 100644 --- a/src/dev/crime_mem.rs +++ b/src/dev/crime_mem.rs @@ -49,11 +49,10 @@ //! ([`RamWindow`]). Every other master reaches memory through the bus. //! //! POST keeps its stack in cache lines tagged for physical page 0 while it -//! tests the same page uncached, so on the machine the two are different bytes. -//! The CPU's cached path and the window always mean the bank, so that is where -//! the stack lives here too, and for as long as those tags stand the decode -//! table points that page at a page of its own for the uncached path. See -//! [`O2Memory::stack_tag_store`]. +//! tests the same page uncached, so on the machine the two are different +//! bytes. The lines live in the CPU's cached view of the window +//! (`PpMemSpace::add_cached_view`), and the bank, the decode table and every +//! other master go on seeing memory. //! //! `O2Memory` is CRIME's bank logic: the sizes fitted, the controls, the //! decode they make, and the remap. Its decode is atomics: the firmware @@ -84,13 +83,6 @@ const SLOTS: usize = (RAM_WINDOW_SIZE / SLOT) as usize; /// A 32 MB slot in the decode table's 4 KB entries. const TABLE_PER_SLOT: u64 = (SLOT >> SLOT_SHIFT) as u64; -/// The one page of physical memory POST keeps its stack in. -const STACK_PAGE: u32 = 4096; -/// TagLo as R10000 and R5000 share it: the primary state, 0 for invalid. -const TAG_STATE: u32 = 0xc0; -/// TagLo's physical page number. -const TAG_PAGE_SHIFT: u32 = 8; - /// The kits SGI sold, in MB per pair. pub const KITS_MB: [u32; 4] = [32, 64, 128, 256]; @@ -179,12 +171,6 @@ pub struct O2Memory { /// What [`RamWindow`]s need to know of the decode, published as it /// changes. shared: Arc, - /// Page 0 as the uncached path sees it while POST's stack is in the bank. - /// It wraps at 4 KB, so the same device answers at every alias. Boxed: the - /// decode table points at it and `O2Memory` is built by value. - stack_page: Box, - /// Whether the decode table is pointing at [`Self::stack_page`]. - stack_live: AtomicBool, } impl O2Memory { @@ -241,8 +227,6 @@ impl O2Memory { } let phys = Physical::new(banks); phys.set_default(Arc::new(NoMemory), "no memory"); - let stack_page = Box::new(RamBank::new(1)); - stack_page.set_addr_mask(STACK_PAGE - 1); let m = Self { phys, size, @@ -253,8 +237,6 @@ impl O2Memory { decode_gen: AtomicU64::new(0), rebuilding: std::sync::Mutex::new(()), shared: Default::default(), - stack_page, - stack_live: AtomicBool::new(false), }; // At reset every control is zero: all eight banks at the bottom of // the window, 32 MB each, so bank 0 answers there. @@ -356,106 +338,10 @@ impl O2Memory { let _ = sp.map_range(i, src, at, SLOT as u64); } } - if self.stack_live.load(Ordering::Relaxed) { - self.place_stack_page(slots, mapped); - } self.shared.decode_gen.store(self.decode_gen.load(Ordering::Acquire), Release); self.shared.mapped.store(mapped, Release); } - /// The decode table's entry for bank 0's offset 0 goes to the stack page, - /// at every place that offset answers: a 32 MB bank under a 128 MB setting - /// repeats, and sizing relies on it. - fn place_stack_page(&self, slots: &[Slot; SLOTS], mapped: u32) { - let phys = &*self.phys; - let page: *const RamBank = &*self.stack_page; - let page = page as crate::ppmem::devmap::Entry; - let size = self.size[0] as u64; - for (s, slot) in slots.iter().enumerate() { - let Slot::Ram { bank: 0, chunk } = *slot else { continue }; - if (chunk as u64 * SLOT as u64) & (size - 1) != 0 { - continue; - } - let at = s as u64 * SLOT as u64; - phys.fill_slots((RAM_BASE as u64 + at) >> SLOT_SHIFT, 1, page); - if at < LOW_ALIAS_SIZE as u64 { - phys.fill_slots(at >> SLOT_SHIFT, 1, page); - } - } - // ppmem's bitmap wants a whole 64 MB region mapped. At reset only the - // first 32 MB slot is, so the cached path would take the bus and find - // the stack page, then find the bank once sizing filled the region. - // Show bank 0 again in the next slot, for the CPU only. - if mapped & 2 == 0 { - if let (Some(sp), Some(i)) = (phys.ppmem_space(), self.index[0]) { - let _ = sp.map_range(i, 0, SLOT as u64, SLOT as u64); - } - } - } - - /// The guest stored `tag` in a primary data cache tag. - /// - /// POST sweeps a page's worth of tags to dirty for physical page 0 and - /// fills them, with nothing read from memory, then writes the same page - /// uncached to test it, and sweeps every tag invalid without a writeback - /// once it has copied what it needs and moved its stack to KSEG1. So a - /// valid tag for page 0 starts the stack and the next invalid tag ends it. - /// IRIX reboots by jumping into the PROM, so this catches a reboot as it - /// does a power-on. Kernels and `sash` store invalid tags only, which is - /// nothing to do here. - pub fn stack_tag_store(&self, tag: u32) { - if tag & TAG_STATE == 0 { - if self.stack_live.load(Ordering::Relaxed) { - self.set_stack_page(false); - } - } else if tag >> TAG_PAGE_SHIFT == 0 && !self.stack_live.load(Ordering::Relaxed) { - self.set_stack_page(true); - } - } - - /// Start the stack (the uncached path gets a copy of the page, the bank - /// goes on holding what the cached path sees) or end it (what was written - /// uncached is what memory holds, as it is when the cache lines go). - pub fn set_stack_page(&self, on: bool) { - let _one = self.rebuilding.lock().unwrap(); - if self.stack_live.load(Ordering::Relaxed) == on { - return; - } - let bank = &self.phys.banks()[self.index[0].expect("bank 0 is fitted")]; - for o in (0..STACK_PAGE as u64).step_by(8) { - if on { - self.stack_page.write64(o, bank.read64(o).data); - } else { - bank.write64(o, self.stack_page.read64(o).data); - } - } - self.stack_live.store(on, Ordering::Relaxed); - self.rebuild_locked(); - } - - /// The stack page's dwords while the stack is live, for a snapshot: the - /// bank is saved with memory and this is what the uncached path sees. - pub fn stack_page_words(&self) -> Option> { - self.stack_live.load(Ordering::Relaxed).then(|| { - (0..STACK_PAGE as u64).step_by(8).map(|o| self.stack_page.read64(o).data).collect() - }) - } - - /// Put back what [`Self::stack_page_words`] gave. The stack is not live - /// beforehand (a machine being restored has been powered on). - pub fn restore_stack_page(&self, words: &[u64]) -> Result<(), String> { - if words.len() * 8 != STACK_PAGE as usize { - return Err(format!("saved stack page is {} dwords", words.len())); - } - let _one = self.rebuilding.lock().unwrap(); - for (k, &w) in words.iter().enumerate() { - self.stack_page.write64(k as u64 * 8, w); - } - self.stack_live.store(true, Ordering::Relaxed); - self.rebuild_locked(); - Ok(()) - } - /// A view of memory through the window, for a master on its own thread /// (the rendering engine). pub fn window(&self) -> RamWindow { @@ -1118,92 +1004,6 @@ mod tests { assert_eq!(w, 0xbeef, "the window agrees"); } - const STACK_TAG: u32 = 0xc8; // dirty, page 0: what POST stores on an R10000 - - /// What each path sees of page 0 once the stack is live, and that memory - /// ends up holding what was written uncached. - #[test] - fn the_stack_and_the_uncached_page_are_different_bytes() { - let m = O2Memory::with_total(64).unwrap(); - m.set_ctrl(0, 0); - let bus = m.phys(); - let base = bus.ppmem_space().unwrap().window_base(); - let cached = |o: usize, base: *mut u8| unsafe { (*(base.add(o) as *const u64)).rotate_left(32) }; - let store = |o: usize, v: u64, base: *mut u8| unsafe { *(base.add(o) as *mut u64) = v.rotate_left(32) }; - assert_eq!(bus.write64(0xff0, 0x1111), BUS_OK); - m.stack_tag_store(0x0); - m.stack_tag_store(STACK_TAG); - assert_eq!(bus.read64(0xff0).data, 0x1111, "the uncached page starts as a copy"); - store(0xff0, 0x2222, base); - assert_eq!(bus.write64(0xff8, 0x3333), BUS_OK); - assert_eq!(cached(0xff0, base), 0x2222, "the stack, cached"); - assert_eq!(cached(0xff8, base), 0, "the test's write does not reach the stack"); - assert_eq!(bus.read64(0xff0).data, 0x1111, "and the stack does not reach the test"); - assert_eq!(bus.read64(0xff8).data, 0x3333); - assert_eq!(bus.read64(RAM_BASE as u64 + 0xff8).data, 0x3333, "through the window's address too"); - assert_eq!(bus.write64(0x1000, 0x4444), BUS_OK); - assert_eq!(cached(0x1000, base), 0x4444, "the next page is memory"); - m.stack_tag_store(0x0); - assert_eq!(cached(0xff8, base), 0x3333, "memory holds what was written uncached"); - assert_eq!(bus.read64(0xff0).data, 0x1111); - store(0xff8, 0x5555, base); - assert_eq!(bus.read64(0xff8).data, 0x5555, "one memory again"); - } - - /// Sizing reads back through a 32 MB bank repeating under a 128 MB - /// setting, so page 0 has to answer the same way at every repeat. - #[test] - fn the_uncached_page_follows_a_small_bank_repeating() { - let m = O2Memory::with_total(32).unwrap(); - m.stack_tag_store(STACK_TAG); - m.set_ctrl(0, CTRL_128M); - put(&m, RAM_BASE + 8, 0xfeed); - assert_eq!(get(&m, RAM_BASE + 3 * SLOT + 8), Some(0xfeed)); - assert_eq!(get(&m, 3 * SLOT + 8), Some(0xfeed), "the low alias"); - m.stack_tag_store(0); - assert_eq!(m.phys().banks()[0].read64(8).data, 0xfeed); - } - - /// At reset one 32 MB slot is mapped, which ppmem's bitmap does not count; - /// the cached path must still reach the bank, not the bus, or the stack - /// moves out from under POST when sizing completes the region. - #[test] - fn the_cached_path_is_on_the_bank_from_reset() { - let m = O2Memory::with_total(32).unwrap(); - let sp = m.phys().ppmem_space().unwrap(); - let region = 1u64 << 0; - assert_eq!(sp.mapped_bitmap() & region, 0, "a hole at reset"); - m.stack_tag_store(STACK_TAG); - assert_ne!(sp.mapped_bitmap() & region, 0, "filled while the stack is live"); - m.stack_tag_store(0); - assert_eq!(sp.mapped_bitmap() & region, 0, "and only then"); - } - - /// Only a valid tag for page 0 starts the stack; other firmware's tags - /// and a kernel's invalid ones leave the decode alone. - #[test] - fn other_tags_leave_the_decode_alone() { - let m = O2Memory::with_total(32).unwrap(); - for tag in [0u32, 0x1_00c8, 0x08] { - m.stack_tag_store(tag); - assert!(m.stack_page_words().is_none(), "{tag:#x}"); - } - } - - /// A snapshot taken during POST comes back with both views. - #[test] - fn a_snapshot_carries_the_uncached_page() { - let m = O2Memory::with_total(32).unwrap(); - m.stack_tag_store(STACK_TAG); - put(&m, 0x20, 0xabcd); - let words = m.stack_page_words().unwrap(); - let n = O2Memory::with_total(32).unwrap(); - n.restore_stack_page(&words).unwrap(); - assert_eq!(get(&n, 0x20), Some(0xabcd)); - n.stack_tag_store(0); - assert_eq!(n.phys().banks()[0].read64(0x20).data, 0xabcd); - } - /// An empty bank's slots read zero; with no control matching, the bus's /// default answers. #[test] diff --git a/src/dev/heart.rs b/src/dev/heart.rs index 61f367de6..58e72cd2e 100644 --- a/src/dev/heart.rs +++ b/src/dev/heart.rs @@ -25,6 +25,7 @@ use parking_lot::Mutex; use crate::cpu::smp::{current_cpu, ParkSlot, HOLD_HEART, MAX_CPUS}; use crate::dev::w1; +use crate::devlog::LogModule; use crate::snapshot::{field, field_bool, field_u32, field_u32s, field_u64, field_u64s, StateTable}; use crate::traits::{BusDevice, BusRead16, BusRead32, BusRead64, BusRead8, Saveable, BUS_ERR, BUS_OK}; @@ -37,6 +38,10 @@ pub const WIDGET_REG_SIZE: u64 = 0x1000; pub const BANKS: usize = 4; const COUNT_MASK: u64 = (1 << 52) - 1; +/// The counter's PIU offset. +const COUNTER: u32 = 0x20000; +/// The `heart` log shows one counter read in this many. +const COUNTER_SAMPLE: u64 = 256; /// The counter ticks at 12.5 MHz. const COUNT_NS: u64 = 80; @@ -214,6 +219,8 @@ pub struct Heart { unsupported_full: AtomicBool, /// Packets that reached HEART's interrupt register, for the monitor. pub packets: AtomicU64, + /// Counter reads while the `heart` log is on, for its sampling. + counter_reads: AtomicU64, } impl Heart { @@ -231,6 +238,7 @@ impl Heart { unsupported: Mutex::new(Vec::new()), unsupported_full: AtomicBool::new(false), packets: AtomicU64::new(0), + counter_reads: AtomicU64::new(0), } } @@ -444,7 +452,7 @@ impl Heart { // The reader's own mask: a CPU services what it enabled. 0x10038 => Some(r.pending & r.mask[current_cpu()]), 0x10040 => Some(r.ecc_cause), - 0x20000 => Some(r.counter((self.clock)())), + COUNTER => Some(r.counter((self.clock)())), 0x30000 => Some(r.compare as u64), // Who is asking. 0x50000 => Some(current_cpu() as u64), @@ -607,6 +615,7 @@ impl Heart { } fn set_vector(&self, vector: u8, set: bool) { + crate::dlog!(LogModule::Heart, "heart: vector {vector} {}", if set { "raised" } else { "lowered" }); self.packets.fetch_add(1, Ordering::Relaxed); let mut r = self.regs.lock(); if set { @@ -658,18 +667,26 @@ impl Heart { } fn bus_read(&self, addr: u64, size: usize) -> Option { - match Self::block(addr)? { - (true, off) => self.piu_read(off, size), - (false, off) => self.widget_read(off, size), + let (piu, off) = Self::block(addr)?; + let v = if piu { self.piu_read(off, size) } else { self.widget_read(off, size) }; + if crate::devlog::devlog_is_active(LogModule::Heart) { + if !(piu && off & !7 == COUNTER) { + crate::devlog::devlog().write(LogModule::Heart, format_args!("heart: read {} {off:#x} /{size} -> {}", + if piu { "piu" } else { "widget" }, v.map_or("bus error".to_string(), |v| format!("{v:#x}")))); + } else if self.counter_reads.fetch_add(1, Ordering::Relaxed) % COUNTER_SAMPLE == 0 { + // The counter is read too often to log every read; one in + // COUNTER_SAMPLE, with the host's clock, shows its rate. + crate::devlog::devlog().write(LogModule::Heart, format_args!("heart: counter read -> {:#x} at host {} ns (1 in {COUNTER_SAMPLE})", + v.unwrap_or(0), (self.clock)())); + } } + v } fn bus_write(&self, addr: u64, size: usize, val: u64) -> u32 { - let ok = match Self::block(addr) { - Some((true, off)) => self.piu_write(off, size, val), - Some((false, off)) => self.widget_write(off, size, val), - None => false, - }; + let Some((piu, off)) = Self::block(addr) else { return BUS_ERR }; + crate::dlog!(LogModule::Heart, "heart: write {} {off:#x} /{size} {val:#x}", if piu { "piu" } else { "widget" }); + let ok = if piu { self.piu_write(off, size, val) } else { self.widget_write(off, size, val) }; if ok { BUS_OK } else { diff --git a/src/dev/ioc3.rs b/src/dev/ioc3.rs index a17803a9b..f7d7ff99f 100644 --- a/src/dev/ioc3.rs +++ b/src/dev/ioc3.rs @@ -14,7 +14,7 @@ //! # Two interrupt outputs //! //! The chip has two. The Ethernet block drives the PCI interrupt pin -//! ([`PciBusDevice::connect_interrupt`]); everything else (serial, PS/2, +//! (the line in its [`PciFunctionState`]); everything else (serial, PS/2, //! parallel, the interrupt-output pulse) is gated by the status and enable //! pair at 0x1c/0x20 and drives a second line the board wires to a Bridge //! input ([`Ioc3::connect_superio_interrupt`]). @@ -24,7 +24,7 @@ //! IOC3 owns the conversion between its register and DMA formats and PCI //! bytes. Registers: a 32-bit access is the register value; a narrow access //! to internal offset `a` arrives from the bridge at `a ^ (4 - width)`, which -//! [`PciBusDevice::bar_read`] undoes. DMA: an internal byte at address `a` +//! [`PciFunction::bar_read`] undoes. DMA: an internal byte at address `a` //! sits at bus address `a ^ 3`, so a 32-bit value is big-endian on the bus //! with the bridge's swap clear ([`Ioc3Dma`]). Neither is a measured property //! of the silicon; the chip's real placement is undocumented. @@ -61,7 +61,7 @@ use crate::dev::w1::sgi::{self, BoardRecord}; use crate::dev::ns16550::Ns16550; use crate::dev::serial_host::SerialBackend; use crate::irq::{InterruptController, IrqLine}; -use crate::pci::{Bar, PciBusDevice, PciConfig, PciConfigPolicy, PciDma, PciDmaStatus, PCI_INTA}; +use crate::pci::{Bar, PciConfig, PciConfigPolicy, PciDma, PciDmaStatus, PciFunction, PciFunctionState, PCI_INTA}; use crate::hptimer::{TimerId, TimerManager, TimerReturn}; use crate::traits::{BusDevice, Device, MachineEvent, Resettable, Saveable}; use crate::snapshot::{ @@ -2106,18 +2106,17 @@ type Deadlines = [Option; KINDS]; type NvramJob = (std::path::PathBuf, Vec); /// Lock order: `gate`, then `line_order`, then `st`; `armed`, the line and -/// Ethernet slots (`pin_line`, `superio_line`, `eth`, `raw_dma`) and the +/// Ethernet slots (`pin_line`, `superio_line`, `eth`, the port's DMA) and the /// NVRAM hand-off are leaves. /// The serial and parallel engines move their rings with `st` held: the /// DMA goes down through the Bridge and HEART, which never call back into /// the chip, and an access to the Ethernet block goes to it with `st` /// released. struct Shared { - pci: PciConfig, + pci: PciFunctionState, st: Mutex, ps2: Arc, eth: Mutex>>, - raw_dma: Mutex>>, /// PCI interrupt pin A: the Ethernet block. pin_line: Mutex>, /// The SuperIO line the board wires to a Bridge input. @@ -2358,7 +2357,7 @@ impl Shared { /// both it and the bridge's DMA are in hand. fn connect_eth(self: &Arc) { let eth = self.eth.lock().clone(); - let dma = self.raw_dma.lock().clone(); + let dma = self.pci.dma().cloned(); if let (Some(e), Some(d)) = (eth, dma) { let weak = Arc::downgrade(self); e.connect( @@ -2445,11 +2444,10 @@ impl Ioc3 { // The subsystem identity selects what a guest instantiates. pci.set_subsystem_identity(((subsystem as u32) << 16) | VENDOR_ID as u32); Shared { - pci, + pci: PciFunctionState::new(pci), st: Mutex::new(St::new(cfg)), ps2, eth: Mutex::new(None), - raw_dma: Mutex::new(None), pin_line: Mutex::new(None), superio_line: Mutex::new(None), heartbeat: OnceLock::new(), @@ -2562,9 +2560,9 @@ impl Ioc3 { self.sh.connect_eth(); } - /// The line the board wires the SuperIO interrupt (serial, PS/2, - /// parallel, interrupt output) to. [`PciBusDevice::connect_interrupt`] - /// takes the PCI pin, which only the Ethernet block drives. + /// The line the board wires the SuperIO interrupt (serial, PS/2, parallel, + /// interrupt output) to. The line in its [`PciFunctionState`] is the PCI + /// pin, which only the Ethernet block drives. pub fn connect_superio_interrupt(&self, line: IrqLine) { { let _order = self.sh.line_order.lock(); @@ -2591,7 +2589,7 @@ impl Ioc3 { /// machine resets it on its own. pub fn reset(&self) { if self.sh.st.lock().cfg.reset_pci_on_power_on { - self.sh.pci.reset(); + self.sh.pci.config().reset(); } self.sh.with_st(false, |st, _| st.reset()); } @@ -2717,13 +2715,13 @@ impl Ioc3 { /// the first dwords of the register block alike. The status register /// reads fast back-to-back capable beside the medium decode timing. fn header_dword(&self, index: usize) -> u32 { - let v = self.sh.pci.read(index); + let v = self.sh.pci.config().read(index); if index == 1 { v | PCI_STATUS_FAST_BACK << 16 } else { v } } } -impl PciBusDevice for Ioc3 { - fn config(&self) -> &PciConfig { +impl PciFunction for Ioc3 { + fn state(&self) -> &PciFunctionState { &self.sh.pci } @@ -2800,20 +2798,21 @@ impl PciBusDevice for Ioc3 { self.sh.with_st(parity, |st, c| st.write(a, val, width, c)); } - /// The PCI interrupt pin: the Ethernet block's level. - fn connect_interrupt(&self, line: IrqLine) { - { - let _order = self.sh.line_order.lock(); - line.set(self.sh.pin_level.load(Ordering::SeqCst)); - *self.sh.pin_line.lock() = Some(line); + /// The chip's DMA goes through its own byte order, and the PCI + /// interrupt pin is the Ethernet block's level. + fn attached(&self) { + if let Some(dma) = self.sh.pci.dma() { + self.sh.st.lock().dma = Some(Arc::new(Ioc3Dma::new(dma.clone()))); + self.sh.connect_eth(); + } + if let Some(line) = self.sh.pci.irq() { + { + let _order = self.sh.line_order.lock(); + line.set(self.sh.pin_level.load(Ordering::SeqCst)); + *self.sh.pin_line.lock() = Some(line.clone()); + } + self.sh.sync_pin(); } - self.sh.sync_pin(); - } - - fn connect_dma(&self, dma: Arc) { - *self.sh.raw_dma.lock() = Some(dma.clone()); - self.sh.st.lock().dma = Some(Arc::new(Ioc3Dma::new(dma))); - self.sh.connect_eth(); } } diff --git a/src/dev/ioc3_tests.rs b/src/dev/ioc3_tests.rs index 825907170..55c99d209 100644 --- a/src/dev/ioc3_tests.rs +++ b/src/dev/ioc3_tests.rs @@ -1,6 +1,6 @@ //! IOC3 register-contract tests. //! -//! Accesses go through `PciBusDevice` the way a host bridge makes them: a +//! Accesses go through `PciFunction` the way a host bridge makes them: a //! 32-bit access at its offset, a byte access at `offset ^ 3`. //! //! Ported from tests/qtest/octane-test.c in jamesbraid/qemu-sgi-origami. @@ -9,7 +9,7 @@ use super::*; use std::sync::Mutex; use crate::irq::LevelProbe; use crate::traits::MachineEvent; -use crate::pci::PciBusDevice; +use crate::pci::PciFunction; const MAC: [u8; 6] = [0x08, 0x00, 0x69, 0x12, 0x34, 0x56]; @@ -78,7 +78,8 @@ impl Rig { fn with(c: Ioc3) -> Rig { let mem = TestMem::new(); - c.connect_dma(mem.clone()); + c.state().connect_dma(mem.clone()); + c.attached(); let wire = [Arc::new(Wire::default()), Arc::new(Wire::default())]; c.set_serial_backend(0, wire[0].clone()); c.set_serial_backend(1, wire[1].clone()); @@ -119,7 +120,7 @@ impl Rig { #[test] fn identity_and_bar() { let r = Rig::new(); - let cfg = r.c.config(); + let cfg = r.c.state().config(); assert_eq!(cfg.read(0), 0x0003_10a9); assert_eq!(cfg.read(2) >> 8, 0xff_0000, "no defined class"); assert_eq!(cfg.read(2) & 0xff, 1, "revision 1"); @@ -141,7 +142,7 @@ fn identity_and_bar() { #[test] fn configuration_space_repeats_its_first_32_bytes() { let r = Rig::new(); - let cfg = r.c.config(); + let cfg = r.c.state().config(); cfg.write(4, 0x0050_0000); cfg.write(3, 0x2800); cfg.write(1, 0x146); @@ -170,7 +171,7 @@ fn configuration_space_repeats_its_first_32_bytes() { #[test] fn the_register_block_starts_with_the_configuration_header() { let r = Rig::new(); - let cfg = r.c.config(); + let cfg = r.c.state().config(); cfg.write(4, 0x0050_0000); cfg.write(3, 0x2800); cfg.write(1, 0x146); @@ -1108,9 +1109,10 @@ fn the_ethernet_level_drives_the_pci_pin_not_the_superio_line() { let c = Ioc3::octane(MAC, false); let eth = Arc::new(MockEth::default()); c.attach_eth(eth.clone()); - c.connect_dma(TestMem::new()); + c.state().connect_dma(TestMem::new()); let (pin, line) = LevelProbe::line(); - c.connect_interrupt(line); + c.state().connect_irq(line); + c.attached(); let (sup, sline) = LevelProbe::line(); c.connect_superio_interrupt(sline); let wired = eth.wired.lock().unwrap(); @@ -1201,9 +1203,10 @@ fn the_ethernet_pin_follows_the_last_level_given() { let c = Ioc3::octane(MAC, false); let eth = Arc::new(MockEth::default()); c.attach_eth(eth.clone()); - c.connect_dma(TestMem::new()); + c.state().connect_dma(TestMem::new()); let (pin, line) = LevelProbe::line(); - c.connect_interrupt(line); + c.state().connect_irq(line); + c.attached(); let wired = eth.wired.lock().unwrap(); let irq = &wired.as_ref().unwrap().1; for i in 0..1000 { @@ -1609,6 +1612,28 @@ fn save_load_round_trip() { } } +/// A restore into a chip that has been in use equals one into a fresh chip. +#[test] +fn ioc3_load_into_a_dirtied_instance_equals_load_into_a_fresh_one() { + let dirty = |r: &Rig, salt: u32| { + r.w8(reg::SERIAL_A + 3, 0x80); + r.w8(reg::SERIAL_A + 7, 4 + salt as u8); + r.w8(reg::SERIAL_A + 3, 0x03); + r.w32(reg::SSRAM + 0x100 * salt, 0x1000 + salt); + r.w32(reg::INT_ENABLE_SET, 0x40 << salt); + }; + let src = Rig::new(); + dirty(&src, 1); + let v = src.c.save_state(); + let used = Rig::new(); + dirty(&used, 2); + let fresh = Rig::new(); + used.c.load_state(&v).unwrap(); + fresh.c.load_state(&v).unwrap(); + assert_eq!(used.c.save_state(), fresh.c.save_state()); + assert_eq!(used.c.save_state(), v); +} + /// A periodic interrupt output keeps the time its next pulse had left, /// counted from the load rather than from the save. #[test] @@ -1642,7 +1667,7 @@ fn a_short_buffer_ram_is_refused() { #[test] fn an_octane_chip_reset_keeps_the_pci_header() { let r = Rig::new(); - let cfg = r.c.config(); + let cfg = r.c.state().config(); cfg.write(1, 0xffff_0146); cfg.write(3, 0xffff_ffff); r.c.reset(); @@ -1658,7 +1683,7 @@ fn the_fuel_population_keeps_its_clock_and_header_policy_across_reset() { let r = Rig::with(Ioc3::new(Ioc3Config::fuel(MAC))); let clock = Arc::new(crate::dev::timekeeper::Timekeeper::fuel_at(2026, 1, 1, 0, 0, 0).unwrap()); r.c.attach_timekeeper(clock.clone()); - let cfg = r.c.config(); + let cfg = r.c.state().config(); cfg.write(1, 0xffff_0146); cfg.write(3, 0xffff_ffff); assert_eq!(cfg.read(1), 0x0280_0146); diff --git a/src/dev/isp1040.rs b/src/dev/isp1040.rs index 1dc60951f..dab58d466 100644 --- a/src/dev/isp1040.rs +++ b/src/dev/isp1040.rs @@ -50,8 +50,9 @@ //! again when the hold ends: an approximation, a chip that had fetched it //! would plausibly move on. A parked entry the guest rewrites meanwhile is //! answered incomplete. A snapshot carries each bus's remaining hold and -//! the parked positions. The ISP12160 would widen `BUSES` and decode the -//! bus from mailbox 2 and its request entries; the per-bus state is there. +//! the parked positions. Only the ISP1040's bus reset starts a hold-off: +//! the ISP12160 (two buses, per-bus state is there) keeps its own bus-reset +//! handling, and its hold-off is not modelled. //! * Only logical unit 0 of a present target exists. A command for another //! unit is passed to the target with that unit number, which answers as //! SCSI says (INQUIRY: no device at this unit; anything else: check @@ -81,7 +82,7 @@ use parking_lot::{Condvar, Mutex, MutexGuard}; use crate::devlog::LogModule; use crate::irq::IrqLine; -use crate::pci::{Bar, PciBusDevice, PciConfig, PciDma, PCI_INTA}; +use crate::pci::{Bar, PciConfig, PciConfigPolicy, PciDma, PciFunction, PciFunctionState, PciRegisterPolicy, PCI_INTA}; use crate::scsi::{scsi_cmd, ScsiBus, ScsiDataLength, ScsiIdentities, ScsiRequest, CDROM_IDENTITY, DISK_IDENTITY}; use crate::snapshot::{get_field, hex_u16, hex_u32, load_u16_slice, toml_u16, toml_u32, toml_u8, u16_slice_to_toml, u8_slice_to_toml}; @@ -115,7 +116,7 @@ pub mod reg { pub const WINDOW_SIZE: u32 = 0x100; } -const MAILBOX_COUNT: usize = 8; +pub(crate) const MAILBOX_COUNT: usize = 8; const MAILBOX_END: u32 = reg::MAILBOX_BASE + MAILBOX_COUNT as u32 * 2; const BANK_END: u32 = 0xc0; const BANK_WORDS: usize = ((BANK_END - reg::BANK_BASE) / 2) as usize; @@ -163,6 +164,8 @@ pub mod mbox { pub const INIT_REQ_QUEUE: u16 = 0x0010; pub const INIT_RSP_QUEUE: u16 = 0x0011; pub const ABORT_COMMAND: u16 = 0x0015; + pub const ABORT_DEVICE: u16 = 0x0016; + pub const ABORT_TARGET: u16 = 0x0017; pub const BUS_RESET: u16 = 0x0018; pub const GET_FIRMWARE_STATUS: u16 = 0x001f; pub const GET_TARGET_PARAMETERS: u16 = 0x0028; @@ -187,13 +190,14 @@ pub mod mbox { pub const INVALID: u16 = 0x4001; pub const HOST_INTERFACE_ERROR: u16 = 0x4002; pub const TEST_FAILED: u16 = 0x4003; + pub const COMMAND_ERROR: u16 = 0x4005; pub const PARAM_ERROR: u16 = 0x4006; } const RAM_WORDS: usize = 0x10000; /// The command budget the adapter announces in its firmware status. pub const MAX_REQUESTS: u16 = 512; -const FW_VERSION: [u16; 3] = [1, 0, 0]; +pub(crate) const FW_VERSION: [u16; 3] = [1, 0, 0]; pub const HOST_CMD_SHIFT: u32 = 12; pub const RISC_FLAG_HOST_INT: u16 = 0x0080; @@ -229,6 +233,7 @@ pub const SCSI_STATUS_SENSE_VALID: u16 = 0x0200; pub const IOCB_COMPLETE: u16 = 0x0000; pub const IOCB_INCOMPLETE: u16 = 0x0001; pub const IOCB_RESET: u16 = 0x0004; +pub const IOCB_ABORTED: u16 = 0x0005; pub const IOCB_DATA_UNDERRUN: u16 = 0x0015; pub const STATE_GOT_BUS: u16 = 0x0100; @@ -256,13 +261,14 @@ const DEFAULT_SELECTION_TIMEOUT: u16 = 250; /// source for a device, so it runs on the adapter's clock, host wall time /// less the time the machine was stopped, like the selection timeout. const BUS_RESET_DELAY_UNIT: Duration = Duration::from_secs(1); -/// The SCSI buses this part drives. The hold-off, the parked commands and the -/// selection are all kept per bus, so a two-bus part widens this and decodes -/// the bus from its own request entries and from mailbox 2. -const BUSES: usize = 1; +/// The most SCSI buses any part drives (the ISP12160's two; the ISP1040 has +/// one, `IspPart::channels`). The hold-off, the parked commands and the +/// selection are all kept per bus. Only the ISP1040's bus reset starts a +/// hold-off: the ISP12160 keeps its own bus-reset handling. +const BUSES: usize = 2; const CHECK_CONDITION: u8 = 0x02; -const MAX_TARGET: u8 = 15; +pub(crate) const MAX_TARGET: u8 = 15; const MAX_LUN: u8 = 7; /// A completion held for response-ring room. @@ -374,6 +380,49 @@ enum Prepared { Run(Job), } +/// Part-specific register and firmware behavior; the ring engine owns the state. +/// Defaults preserve the ISP1040 contract, including wrapping firmware RAM. +pub(crate) trait IspPart: Send + Sync { + fn name(&self) -> &'static str; + fn channels(&self) -> usize { 1 } + fn pci_config(&self, bars: [Bar; 6]) -> PciConfig { + PciConfig::new(PCI_VENDOR_QLOGIC, PCI_DEVICE_ISP1040, PCI_CLASS_SCSI, PCI_REVISION_ISP1040, bars, PCI_INTA) + } + fn firmware_version(&self) -> [u16; 3] { FW_VERSION } + fn active_bank(&self, cfg1: u16) -> Bank { + if cfg1 & CFG1_BANK_SELECT != 0 { Bank::Sxp } else { Bank::Risc } + } + fn checks_ram_bounds(&self) -> bool { false } + fn has_abort_commands(&self) -> bool { false } + fn target_on(&self, _channel: u8, target: u8) -> u8 { target } + fn decode_target(&self, target: u8) -> (usize, u8) { (0, target) } + fn id_high(&self) -> u16 { PCI_DEVICE_ISP1040 } + /// The ISP1040 register revision runs one ahead of its PCI revision. + fn bus_cfg0(&self) -> u16 { PCI_REVISION_ISP1040 as u16 + 1 } + fn cfg1_write_mask(&self) -> u16 { CFG1_WRITE_MASK } + fn unmodelled_dma_register(&self, _off: u32) -> bool { false } + fn completion_phases(&self, _moved: u64, _total: u64) -> u16 { STATE_ALL_PHASES & !STATE_GOT_SENSE } + fn mailbox_ready(&self, _semaphore: u16) -> bool { true } + /// Extra target-parameter results, beyond the common asynchronous/narrow reply. + fn clear_target_parameters(&self, _mailbox: &mut [u16; MAILBOX_COUNT]) {} + fn set_reset_delay(&self, st: &mut State) { st.reset_delay = st.mailbox[1]; } + /// Queue target notifications for the worker; the CPU never locks a bus. + fn reset_targets(&self, _st: &mut State) -> bool { false } + /// Handle extra mailbox commands, leaving result signalling to the engine. + fn execute_mailbox(&self, _core: &Core, _st: &mut MutexGuard<'_, State>) -> bool { false } + fn save_extra(&self, _st: &State, _t: &mut toml::map::Map) {} + fn load_extra(&self, _st: &mut State, _v: &toml::Value) {} +} + +struct Isp1040Part; + +impl IspPart for Isp1040Part { + fn name(&self) -> &'static str { "ISP1040" } +} + +/// The register window selected by BUS_CFG1, independent of its encoding. +pub(crate) enum Bank { Risc, Sxp, Sxp1, Dma } + /// A command waiting out its bus's post-reset hold-off. Only the ring /// position is kept: the entry, its data segments included, is read again /// from the guest's ring when the hold ends. @@ -392,7 +441,15 @@ struct Parked { started: bool, } -struct State { +pub(crate) struct State { + pub(crate) reset_targets: bool, + pub(crate) pending_target_resets: VecDeque<(u8, Option)>, + pub(crate) active: Option<(u8, u8, u32)>, + cancelled: bool, + pub(crate) outstanding: Vec<(u8, u8, u32)>, + pub(crate) sxp1_bank: [u16; BANK_WORDS], + pub(crate) dma_bank: [u16; BANK_WORDS], + pub(crate) reset_delay1: u16, ctl: Ctl, bus_cfg1: u16, bus_ctrl: u16, @@ -402,11 +459,11 @@ struct State { risc_flags: u16, dma_cmd_cfg: u16, dma_data_cfg: u16, - mailbox: [u16; MAILBOX_COUNT], + pub(crate) mailbox: [u16; MAILBOX_COUNT], risc_bank: [u16; BANK_WORDS], sxp_bank: [u16; BANK_WORDS], risc_ram: Vec, - reset_delay: u16, + pub(crate) reset_delay: u16, /// Mailbox command 0x0031's value, in [`SELECTION_TIMEOUT_UNIT`]s. selection_timeout: u16, rqst_base: u64, @@ -434,6 +491,14 @@ struct State { impl State { fn new() -> Self { let mut s = State { + reset_targets: false, + pending_target_resets: VecDeque::new(), + active: None, + cancelled: false, + outstanding: Vec::new(), + sxp1_bank: [0; BANK_WORDS], + dma_bank: [0; BANK_WORDS], + reset_delay1: 0, ctl: Ctl::default(), bus_cfg1: 0, bus_ctrl: 0, @@ -485,6 +550,12 @@ impl State { self.mailbox = [0, SIG1, SIG2, SIG3, SIG4, 0, 0, 0]; self.risc_bank = [0; BANK_WORDS]; self.sxp_bank = [0; BANK_WORDS]; + self.sxp1_bank = [0; BANK_WORDS]; + self.dma_bank = [0; BANK_WORDS]; + self.reset_delay1 = 0; + self.active = None; + self.cancelled = false; + self.outstanding.clear(); // The driver re-downloads firmware after every reset. self.risc_ram.iter_mut().for_each(|w| *w = 0); self.reset_delay = 0; @@ -499,6 +570,8 @@ impl State { self.resp_seen = 0; self.resp_cons = 0; self.pending.clear(); + self.reset_targets = false; + self.pending_target_resets.clear(); self.quiet_until = [None; BUSES]; self.parked.clear(); } @@ -524,9 +597,15 @@ impl State { self.resp_len != 0 && (u32::from(self.resp_prod) + 1) % u32::from(self.resp_len) != u32::from(self.resp_cons) } - fn active_bank(&mut self) -> &mut [u16; BANK_WORDS] { - if self.bus_cfg1 & CFG1_BANK_SELECT != 0 { &mut self.sxp_bank } else { &mut self.risc_bank } + fn active_bank(&mut self, bank: Bank) -> &mut [u16; BANK_WORDS] { + match bank { + Bank::Risc => &mut self.risc_bank, + Bank::Sxp => &mut self.sxp_bank, + Bank::Sxp1 => &mut self.sxp1_bank, + Bank::Dma => &mut self.dma_bank, + } } + } /// The adapter. Its state lives in a [`Core`] shared with the thread that @@ -543,8 +622,8 @@ impl std::ops::Deref for Isp1040 { /// Lock order: `line_order`, then `st`. Nothing else is taken with `st` /// held except the leaf locks (`notes`, `dma_misses`, `odd_accesses`): DMA, -/// the targets and `fence` are reached with it released. `fence` is held -/// across one guest-memory store at a time, and `targets` across one +/// either bus and `fence` are reached with it released. `fence` is held +/// across one guest-memory store at a time, and either bus across one /// target command; neither is held with the other. pub struct Core { st: Mutex, @@ -560,20 +639,22 @@ pub struct Core { wake: Condvar, /// Serialises driving the interrupt line; see [`Core::sync_irq`]. line_order: Mutex<()>, - /// The targets on the bus, by SCSI id. - bus: ScsiBus, - /// A bus reset the targets have not yet seen. `st` and `bus` are never - /// held together, so the reset (taken under `st`) is passed on by the - /// next command to take `bus`, before it runs: no command can see the - /// targets as they were. See [`Core::scsi`]. - targets_reset: AtomicBool, - dma: OnceLock>, - irq: OnceLock, + /// Generation for commands cancelled by aborts without resetting the chip. + cancel: AtomicU64, + /// Immutable part policy, shared by register, worker and snapshot paths. + part: &'static dyn IspPart, + /// The controller's one bus, or both channels of the ISP12160. + buses: [ScsiBus; 2], + /// A bus reset the targets on that bus have not yet seen. `st` and the + /// buses are never held together, so the reset (taken under `st`) is + /// passed on by the next command to take the bus, before it runs: no + /// command can see the targets as they were. See [`Core::scsi`]. + targets_reset: [AtomicBool; 2], /// Which of the Octane's two controllers this is (0 internal, 1 /// external): the monitor's `scsi`/`cow` address the first, /// `scsi1`/`cow1` the second. index: u8, - pci: PciConfig, + pci: PciFunctionState, notes: Mutex>, dma_misses: Mutex>, odd_accesses: Mutex, @@ -603,7 +684,11 @@ impl Isp1040 { /// The controller at `index` (0 internal, 1 external) on `bus`. pub fn new_at(index: u8, bus: ScsiBus) -> Self { - Self { core: Arc::new(Core::new_at(index, bus)), worker: Mutex::new(None) } + Self { core: Arc::new(Core::new_at(index, &Isp1040Part, [bus, Self::new_bus()])), worker: Mutex::new(None) } + } + + pub(crate) fn new_12160(bus0: ScsiBus, bus1: ScsiBus) -> Self { + Self { core: Arc::new(Core::new_at(0, &crate::dev::isp12160::Isp12160Part, [bus0, bus1])), worker: Mutex::new(None) } } /// Start the ring thread (idempotent) and let it take whatever was @@ -617,7 +702,7 @@ impl Isp1040 { let c = self.core.clone(); *worker = Some( std::thread::Builder::new() - .name(format!("ISP1040-{}", self.index)) + .name(format!("{}-{}", self.core.model_name(), self.index)) .spawn(move || c.run_ring()) .expect("spawn isp1040 thread"), ); @@ -641,13 +726,17 @@ impl Isp1040 { /// Name the adapter's long-lived locks in the lock registry, which /// holds only a weak reference to them. pub fn register_locks(&self) { - for (what, probe) in [ + let mut locks: Vec<(&str, fn(&Core) -> bool)> = vec![ ("st", (|c: &Core| c.st.is_locked()) as fn(&Core) -> bool), ("fence", |c| c.fence.is_locked()), - ("targets", |c| c.bus.is_locked()), - ] { + ("targets", |c| c.buses[0].is_locked()), + ]; + if self.core.part.channels() > 1 { + locks.push(("targets1", |c| c.buses[1].is_locked())); + } + for (what, probe) in locks { let core = Arc::downgrade(&self.core); - crate::locks::register_lock(format!("isp1040-{}::{what}", self.index), move || { + crate::locks::register_lock(format!("{}-{}::{what}", self.core.model_name().to_ascii_lowercase(), self.index), move || { core.upgrade().is_some_and(|c| probe(&c)) }); } @@ -662,27 +751,25 @@ impl Drop for Isp1040 { } impl Core { - fn new_at(index: u8, bus: ScsiBus) -> Self { + fn new_at(index: u8, part: &'static dyn IspPart, buses: [ScsiBus; 2]) -> Self { let st = State::new(); let gen = st.ctl.gen; + let bars = [Bar::Io(reg::WINDOW_SIZE), Bar::Mem(reg::WINDOW_SIZE), Bar::Unused, Bar::Unused, Bar::Unused, Bar::Unused]; + let pci = part.pci_config(bars); Self { + part, + cancel: AtomicU64::new(0), st: Mutex::new(st), gen: AtomicU64::new(gen), fence: Mutex::new(()), wake: Condvar::new(), line_order: Mutex::new(()), - bus, - targets_reset: AtomicBool::new(false), - dma: OnceLock::new(), - irq: OnceLock::new(), + buses, + targets_reset: [AtomicBool::new(false), AtomicBool::new(false)], index, // BAR 0 is the I/O window and BAR 1 its memory-space alias, one // register file. - pci: PciConfig::new( - PCI_VENDOR_QLOGIC, PCI_DEVICE_ISP1040, PCI_CLASS_SCSI, PCI_REVISION_ISP1040, - [Bar::Io(reg::WINDOW_SIZE), Bar::Mem(reg::WINDOW_SIZE), Bar::Unused, Bar::Unused, Bar::Unused, Bar::Unused], - PCI_INTA, - ), + pci: PciFunctionState::new(pci), notes: Mutex::new(Vec::new()), dma_misses: Mutex::new(std::collections::BTreeSet::new()), odd_accesses: Mutex::new(0), @@ -692,6 +779,8 @@ impl Core { } } + fn model_name(&self) -> &'static str { self.part.name() } + pub fn index(&self) -> u8 { self.index } /// A machine reset: the chip as at power-on (no firmware, no queues), @@ -702,6 +791,7 @@ impl Core { { let mut st = self.st.lock(); st.reset(); + if self.part.reset_targets(&mut st) { self.kick(&mut st); } self.publish_gen(&st); self.wake.notify_all(); } @@ -731,12 +821,6 @@ impl Core { let _ = self.heartbeat.set(heartbeat); } - /// Wire the adapter's interrupt pin to `line` (a level). - pub fn connect_irq(&self, line: IrqLine) { - let _ = self.irq.set(line); - self.sync_irq(); - } - /// True while the adapter asserts its interrupt line. pub fn interrupting(&self) -> bool { self.st.lock().irq_level() @@ -749,7 +833,7 @@ impl Core { fn sync_irq(&self) { let _order = self.line_order.lock(); let level = self.st.lock().irq_level(); - if let Some(l) = self.irq.get() { + if let Some(l) = self.pci.irq() { l.set(level); } } @@ -929,7 +1013,7 @@ impl Core { } /// Tell the ring thread the ring may have moved. - fn kick(&self, st: &mut State) { + pub(crate) fn kick(&self, st: &mut State) { st.ctl.work = true; self.wake.notify_all(); } @@ -957,11 +1041,16 @@ impl Core { /// The SCSI bus: attach targets to it and ask what is on it. pub fn bus(&self) -> &ScsiBus { - &self.bus + &self.buses[0] } - fn has_target(&self, id: u8) -> bool { - self.bus.has_target(id) + pub(crate) fn bus_for_channel(&self, channel: u8) -> Option<&ScsiBus> { + ((channel as usize) < self.part.channels()).then(|| &self.buses[channel as usize]) + } + + fn bus_for_target(&self, target: u8) -> (&ScsiBus, u8) { + let (channel, id) = self.part.decode_target(target); + (&self.buses[channel], id) } // ---- DMA --------------------------------------------------------- @@ -985,7 +1074,7 @@ impl Core { /// `buf.len()` bytes from bus address `addr`; false (and zeros) if the /// fabric refuses any of them. fn dma_read(&self, addr: u64, buf: &mut [u8]) -> bool { - let ok = Self::dma_ok(addr, buf.len()) && self.dma.get().is_some_and(|d| d.dma_read(addr, buf).is_ok()); + let ok = Self::dma_ok(addr, buf.len()) && self.pci.dma().is_some_and(|d| d.dma_read(addr, buf).is_ok()); if !ok { buf.iter_mut().for_each(|b| *b = 0); self.dma_miss(addr, false); @@ -994,7 +1083,7 @@ impl Core { } fn dma_write(&self, addr: u64, data: &[u8]) -> bool { - let ok = Self::dma_ok(addr, data.len()) && self.dma.get().is_some_and(|d| d.dma_write(addr, data).is_ok()); + let ok = Self::dma_ok(addr, data.len()) && self.pci.dma().is_some_and(|d| d.dma_write(addr, data).is_ok()); if !ok { self.dma_miss(addr, true); } @@ -1045,12 +1134,18 @@ impl Core { /// A RISC RAM load or dump of up to 128 KiB, with the state unlocked /// for the DMA. The whole transfer is read before a word of it is kept, /// so a refused DMA leaves the array as it was. Words travel - /// little-endian. This part has no edge to its array: a transfer past - /// the end folds to the start. - fn ram_transfer(&self, st: &mut MutexGuard<'_, State>, load: bool) -> u16 { + /// little-endian. The ISP1040 wraps past the end; the ISP12160 rejects + /// transfers that would cross it. + pub(crate) fn ram_transfer(&self, st: &mut MutexGuard<'_, State>, load: bool) -> u16 { let base = st.mailbox[1]; let count = st.mailbox[4] as usize; - let host = Self::mbox_host_addr(&st.mailbox); + if self.part.checks_ram_bounds() && base as usize + count > RAM_WORDS { + return mbox::PARAM_ERROR; + } + let mut host = Self::mbox_host_addr(&st.mailbox); + if st.mailbox[0] == mbox::LOAD_RAM_A64 { + host |= ((st.mailbox[7] as u64) << 32) | ((st.mailbox[6] as u64) << 48); + } if count == 0 { return mbox::COMPLETE; } @@ -1077,6 +1172,20 @@ impl Core { /// Synchronous: the protocol has no way to leave a command pending. fn mailbox_command(&self, st: &mut MutexGuard<'_, State>) { let cmd = st.mailbox[0]; + if self.part.has_abort_commands() && matches!(cmd, mbox::ABORT_COMMAND | mbox::ABORT_DEVICE | mbox::ABORT_TARGET | mbox::BUS_RESET) { + self.with_dispatch_held(st, |st| self.execute_mailbox_command(st)); + } else { + self.execute_mailbox_command(st); + } + } + + fn execute_mailbox_command(&self, st: &mut MutexGuard<'_, State>) { + let cmd = st.mailbox[0]; + if self.part.execute_mailbox(self, st) { + st.semaphore = SEMAPHORE_LOCK; + st.bus_status |= STATUS_INT_REQ | STATUS_RISC_INT_REQ; + return; + } match cmd { mbox::NOP => st.mailbox[0] = mbox::COMPLETE, mbox::WRITE_RAM_WORD => { @@ -1098,7 +1207,19 @@ impl Core { // zero. mbox::VERIFY_CHECKSUM => { let start = st.mailbox[1]; + if self.part.checks_ram_bounds() && start as usize + 4 > RAM_WORDS { + st.mailbox[0] = mbox::PARAM_ERROR; + st.semaphore = SEMAPHORE_LOCK; + st.bus_status |= STATUS_INT_REQ | STATUS_RISC_INT_REQ; + return; + } let len = st.risc_ram[start.wrapping_add(3) as usize]; + if self.part.checks_ram_bounds() && start as usize + len as usize > RAM_WORDS { + st.mailbox[0] = mbox::PARAM_ERROR; + st.semaphore = SEMAPHORE_LOCK; + st.bus_status |= STATUS_INT_REQ | STATUS_RISC_INT_REQ; + return; + } let mut sum = 0u16; for i in 0..len { sum = sum.wrapping_add(st.risc_ram[start.wrapping_add(i) as usize]); @@ -1111,9 +1232,7 @@ impl Core { // Accepted, never executed. mbox::EXEC_FIRMWARE => st.mailbox[0] = mbox::COMPLETE, mbox::ABOUT_FIRMWARE => { - st.mailbox[1] = FW_VERSION[0]; - st.mailbox[2] = FW_VERSION[1]; - st.mailbox[3] = FW_VERSION[2]; + st.mailbox[1..4].copy_from_slice(&self.part.firmware_version()); st.mailbox[6] = 0; st.mailbox[0] = mbox::COMPLETE; } @@ -1128,6 +1247,7 @@ impl Core { mbox::GET_TARGET_PARAMETERS => { st.mailbox[2] = 0; st.mailbox[3] = 0; + self.part.clear_target_parameters(&mut st.mailbox); st.mailbox[0] = mbox::COMPLETE; } // Setters for state nothing reads back; answering invalid is a @@ -1149,7 +1269,7 @@ impl Core { st.mailbox[0] = mbox::COMPLETE; } mbox::SET_RESET_DELAY_PARAMS => { - st.reset_delay = st.mailbox[1]; + self.part.set_reset_delay(st); st.mailbox[0] = mbox::COMPLETE; } mbox::INIT_REQ_QUEUE | mbox::INIT_REQ_QUEUE_A64 | mbox::INIT_RSP_QUEUE | mbox::INIT_RSP_QUEUE_A64 => { @@ -1204,7 +1324,7 @@ impl Core { mbox::BUS_RESET => { let now = self.now_ns(st); let bus = usize::from(st.mailbox[2]); - if bus >= BUSES { + if bus >= self.part.channels() { // A bus this part does not have: nothing to quiet or // retire, and the datasheet's per-bus hold leaves the // real bus alone. Noted so a driver that leaves mailbox 2 @@ -1228,7 +1348,7 @@ impl Core { st.pending.push_back(Pending { handle: p.handle, completion: IOCB_RESET, ..Default::default() }); } } - self.targets_reset.store(true, Ordering::Release); + self.targets_reset[bus].store(true, Ordering::Release); let delay = BUS_RESET_DELAY_UNIT * u32::from(st.mailbox[1]); st.quiet_until[bus] = (!delay.is_zero()).then(|| now.saturating_add(delay.as_nanos() as i64)); self.kick(st); @@ -1245,6 +1365,67 @@ impl Core { st.bus_status |= STATUS_INT_REQ | STATUS_RISC_INT_REQ; } + /// Prevent the worker from taking another entry while a mailbox drops `st`. + fn with_dispatch_held(&self, st: &mut MutexGuard<'_, State>, f: impl FnOnce(&mut MutexGuard<'_, State>) -> R) -> R { + struct Hold<'a, 'b> { + core: &'a Core, + st: &'a mut MutexGuard<'b, State>, + } + impl Drop for Hold<'_, '_> { + fn drop(&mut self) { + self.st.ctl.hold -= 1; + self.core.kick(self.st); + } + } + st.ctl.hold += 1; + let held = Hold { core: self, st }; + f(held.st) + } + + /// A reset of `channel`'s bus: each disk on it puts its mode parameters back before the next + /// command, see [`ScsiTarget::reset_mode_parameters`]. A channel the part lacks is ignored. + pub(crate) fn note_bus_reset(&self, channel: usize) { + if let Some(flag) = self.targets_reset.get(channel) { flag.store(true, Ordering::Release); } + } + + /// Notify targets with `st` released; a blocked target must not stall register access. + fn reset_targets(&self, st: &mut MutexGuard<'_, State>, channel: u8, target: Option) { + let Some(bus) = self.bus_for_channel(channel) else { return }; + self.with_dispatch_held(st, |st| MutexGuard::unlocked(st, || bus.for_each(|id, dev| { + if target.is_none_or(|target| target as usize == id) { + dev.bus_reset(); + } + }))); + } + + pub(crate) fn retire_outstanding(&self, st: &mut State, matches: impl Fn((u8, u8, u32)) -> bool, completion: u16) -> usize { + let mut retired = 0; + let mut keep = Vec::new(); + for item in st.outstanding.drain(..) { + if matches(item) { + st.pending.push_back(Pending { handle: item.2, completion, ..Pending::default() }); + retired += 1; + } else { + keep.push(item); + } + } + st.outstanding = keep; + if retired != 0 { self.kick(st); } + retired + } + + /// Queue one cancellation reply, then fence its remaining guest-memory stores. + pub(crate) fn cancel_active(&self, st: &mut MutexGuard<'_, State>, completion: u16) { + if st.cancelled { return; } + if let Some((_, _, handle)) = st.active { + self.cancel.fetch_add(1, Ordering::AcqRel); + st.cancelled = true; + st.pending.push_back(Pending { handle, completion, ..Pending::default() }); + self.kick(st); + MutexGuard::unlocked(st, || self.retire_stores()); + } + } + fn host_command(&self, st: &mut MutexGuard<'_, State>, val: u16) { match val >> HOST_CMD_SHIFT { 0x0 => {} @@ -1255,6 +1436,10 @@ impl Core { 0x3 => st.risc_flags &= !(RISC_FLAG_PAUSED | RISC_FLAG_RESET), // "A mailbox command is ready": the trigger to process one. 0x5 => { + if !self.part.mailbox_ready(st.semaphore) { + self.note("mailbox submitted with an unacknowledged result".into()); + return; + } st.risc_flags |= RISC_FLAG_HOST_INT; self.mailbox_command(st); } @@ -1440,17 +1625,19 @@ impl Core { /// Run a CDB against the target; `data` is what a write sends it. None /// when a reset cancelled the command while it waited for the target. - fn scsi(&self, gen: u64, target: u8, cdb: &[u8], data: Option>) -> Option<(Vec, u8)> { - let mut targets = self.bus.lock(); - if self.targets_reset.swap(false, Ordering::AcqRel) { + fn scsi(&self, gen: u64, cancel: u64, target: u8, cdb: &[u8], data: Option>) -> Option<(Vec, u8)> { + let (channel, _) = self.part.decode_target(target); + let (bus, id) = self.bus_for_target(target); + let mut targets = bus.lock(); + if self.targets_reset[channel].swap(false, Ordering::AcqRel) { // The bus reset puts each target's mode parameters back; see // [`ScsiTarget::reset_mode_parameters`]. targets.iter_mut().flatten().for_each(|t| t.reset_mode_parameters()); } - if self.gen.load(Ordering::Acquire) != gen { + if self.gen.load(Ordering::Acquire) != gen || self.cancel.load(Ordering::Acquire) != cancel { return None; } - let Some(dev) = targets.get_mut(target as usize).and_then(|t| t.as_mut()) else { + let Some(dev) = targets.get_mut(id as usize).and_then(|t| t.as_mut()) else { return Some((Vec::new(), CHECK_CONDITION)); }; let req = ScsiRequest { cdb: cdb.to_vec(), data_len: ScsiDataLength::Unlimited, data_in: data }; @@ -1481,14 +1668,15 @@ impl Core { let lun = (device & 0xff) as u8; // A bus this part does not have. - if usize::from(channel) >= BUSES { + if channel as usize >= self.part.channels() { return Prepared::Answered(Self::incomplete(handle, 0, 0)); } // GOT_BUS alone is Linux's bad-target; the timeout bit beside it is // what lets the empty slot be treated as benign. - if target > MAX_TARGET || !self.has_target(target) { + if target > MAX_TARGET || !self.buses[channel as usize].has_target(target) { return Prepared::Absent(channel as u8, Self::incomplete(handle, STATE_GOT_BUS, STATUS_FLAG_TIMEOUT)); } + let id = target; if lun > MAX_LUN { return Prepared::Answered(Self::incomplete(handle, STATE_GOT_BUS, STATUS_FLAG_TIMEOUT)); } @@ -1497,9 +1685,9 @@ impl Core { cdb[1] = (cdb[1] & 0x1f) | (lun << 5); let cdb = &cdb[..(cdb_len as usize).clamp(1, CDB_FIELD_LEN)]; let block = self - .bus + .buses[channel as usize] .lock() - .get(target as usize) + .get(id as usize) .and_then(|t| t.as_ref()) .and_then(|t| t.block_size()) .map(|b| b as usize) @@ -1507,6 +1695,7 @@ impl Core { .unwrap_or(BLOCK_SIZE); let (xfer, to_dev) = Self::cdb_transfer(cdb, block); let (segs, described) = self.read_segments(ring, addr, entry_count, seg_count, xfer as u64, a64); + let target = self.part.target_on(channel as u8, target); Prepared::Run(Job { handle, target, cdb: cdb.to_vec(), xfer, to_dev, segs, described }) } @@ -1522,7 +1711,7 @@ impl Core { /// None too when a reset cancelled the command (`gen` is the generation /// it was read under): what it had not yet stored in host memory is not /// stored, and no completion is built. - fn run_job(&self, job: Job, gen: u64) -> Option { + fn run_job(&self, job: Job, gen: u64, cancel: u64) -> Option { let Job { handle, target, cdb, xfer, to_dev, segs, described } = job; // The bar has lights for SCSI ids 0-5. if let (Some(hb), 0..=5) = (self.heartbeat.get(), target) { @@ -1540,9 +1729,10 @@ impl Core { buf.extend_from_slice(&chunk); } moved = buf.len() as u64; - status = self.scsi(gen, target, &cdb, Some(buf))?.1; + if self.cancel.load(Ordering::Acquire) != cancel { return None; } + status = self.scsi(gen, cancel, target, &cdb, Some(buf))?.1; } else { - let (data, s) = self.scsi(gen, target, &cdb, None)?; + let (data, s) = self.scsi(gen, cancel, target, &cdb, None)?; status = s; // Offer the target's bytes to the descriptors in order. let take = (data.len() as u64).min(described).min(xfer as u64) as usize; @@ -1550,7 +1740,7 @@ impl Core { for &(a, l) in &segs { if off >= take { break; } let n = (l as usize).min(take - off); - if !self.fenced(gen, || self.dma_write(a, &data[off..off + n]))? { + if !self.fenced(gen, || self.cancel.load(Ordering::Acquire) == cancel && self.dma_write(a, &data[off..off + n]))? { self.note(format!("payload DMA for handle {handle:#x} refused by the fabric; command stopped")); return None; } @@ -1561,12 +1751,13 @@ impl Core { // Auto-sense runs only after a CHECK CONDITION. let mut sense = Vec::new(); + if self.cancel.load(Ordering::Acquire) != cancel { return None; } if status == CHECK_CONDITION { - sense = self.scsi(gen, target, &[scsi_cmd::REQUEST_SENSE, 0, 0, 0, SENSE_MAX as u8, 0], None)?.0; + sense = self.scsi(gen, cancel, target, &[scsi_cmd::REQUEST_SENSE, 0, 0, 0, SENSE_MAX as u8, 0], None)?.0; sense.truncate(SENSE_MAX); } let mut scsi_status = status as u16; - let mut state_flags = STATE_ALL_PHASES & !STATE_GOT_SENSE; + let mut state_flags = self.part.completion_phases(moved, described.max(xfer as u64)); if !sense.is_empty() { scsi_status |= SCSI_STATUS_SENSE_VALID; state_flags |= STATE_GOT_SENSE; @@ -1663,6 +1854,17 @@ impl Core { st.ctl.work = true; return st; } + if std::mem::take(&mut st.reset_targets) { + for channel in 0..self.part.channels() { + self.note_bus_reset(channel); + self.reset_targets(&mut st, channel as u8, None); + } + continue; + } + if let Some((channel, target)) = st.pending_target_resets.pop_front() { + self.reset_targets(&mut st, channel, target); + continue; + } // Completions held for room go first, in order, as the driver // frees it; while any is held the request ring waits. if !st.pending.is_empty() { @@ -1804,8 +2006,16 @@ impl Core { } Prepared::Run(job) => { st.total_cmds += 1; + st.active = Some((job.target, job.cdb.get(1).copied().unwrap_or(0) >> 5, job.handle)); + st.cancelled = false; st.ctl.busy = true; - let done = MutexGuard::unlocked(&mut st, || self.run_job(job, ring.gen)); + let cancel = self.cancel.load(Ordering::Acquire); + let mut done = MutexGuard::unlocked(&mut st, || self.run_job(job, ring.gen, cancel)); + if st.cancelled { done = None; } + else if self.part.has_abort_commands() && done.is_none() && st.ctl.gen == ring.gen { + if let Some(active) = st.active { st.outstanding.push(active); } + } + st.active = None; st.ctl.busy = false; self.wake.notify_all(); // A reset cancelled the command. @@ -1838,13 +2048,16 @@ impl Core { return st.mailbox[((off - reg::MAILBOX_BASE) / 2) as usize]; } if (reg::BANK_BASE..BANK_END).contains(&off) { - return st.active_bank()[((off - reg::BANK_BASE) / 2) as usize]; + let bank = self.part.active_bank(st.bus_cfg1); + return st.active_bank(bank)[((off - reg::BANK_BASE) / 2) as usize]; + } + if self.part.unmodelled_dma_register(off) { + return 0; } match off { reg::ID_LOW => PCI_VENDOR_QLOGIC, - reg::ID_HIGH => PCI_DEVICE_ISP1040, - // The revision field runs one ahead of the PCI byte. - reg::BUS_CFG0 => PCI_REVISION_ISP1040 as u16 + 1, + reg::ID_HIGH => self.part.id_high(), + reg::BUS_CFG0 => self.part.bus_cfg0(), reg::BUS_CFG1 => st.bus_cfg1, reg::BUS_CTRL => st.bus_ctrl, reg::BUS_STATUS => st.bus_status, @@ -1862,6 +2075,10 @@ impl Core { } fn reg_write(&self, off: u32, val: u16) { + if self.part.unmodelled_dma_register(off) { + self.note(format!("write to ISP12160 DMA register {off:#04x} is not modelled")); + return; + } let mut st = self.st.lock(); if off == reg::RESPONSE_QUEUE { // The consumer acknowledgement: recorded, never over the @@ -1884,15 +2101,17 @@ impl Core { return; } if (reg::BANK_BASE..BANK_END).contains(&off) { - st.active_bank()[((off - reg::BANK_BASE) / 2) as usize] = val; + let bank = self.part.active_bank(st.bus_cfg1); + st.active_bank(bank)[((off - reg::BANK_BASE) / 2) as usize] = val; return; } match off { reg::BUS_CFG1 => { - if val & !CFG1_WRITE_MASK != 0 { - self.note(format!("bus configuration 1 bits {:#06x} are not modelled", val & !CFG1_WRITE_MASK)); + let mask = self.part.cfg1_write_mask(); + if val & !mask != 0 { + self.note(format!("bus configuration 1 bits {:#06x} are not modelled", val & !mask)); } - st.bus_cfg1 = val & CFG1_WRITE_MASK; + st.bus_cfg1 = val & mask; } // Applied after the enables are stored, so the reset wins. reg::BUS_CTRL => { @@ -1900,6 +2119,7 @@ impl Core { // The command in flight is cancelled, not waited for. if val & CTRL_SOFT_RESET != 0 { st.reset(); + if self.part.reset_targets(&mut st) { self.kick(&mut st); } self.publish_gen(&st); self.wake.notify_all(); drop(st); @@ -2005,13 +2225,13 @@ impl crate::traits::Device for Isp1040 { impl crate::scsi_monitor::ScsiTargets for Isp1040 { fn scsi_bus(&self) -> &ScsiBus { - &self.bus + &self.buses[0] } } /// Both BARs reach the same registers. -impl PciBusDevice for Isp1040 { - fn config(&self) -> &PciConfig { +impl PciFunction for Isp1040 { + fn state(&self) -> &PciFunctionState { &self.pci } @@ -2023,12 +2243,13 @@ impl PciBusDevice for Isp1040 { self.write(off, val, width) } - fn connect_interrupt(&self, line: IrqLine) { - self.connect_irq(line); + /// Drive the pin the bus wired to the adapter's state. + fn attached(&self) { + self.sync_irq(); } - fn connect_dma(&self, dma: Arc) { - let _ = self.dma.set(dma); + fn reset(&self) { + self.core.reset(); } } @@ -2041,14 +2262,15 @@ impl crate::traits::Saveable for Isp1040 { // A hold-off already over but not acted on still crosses: the // commands parked behind it start when the load re-arms it at 0. let quiet_left = st.quiet_until.map(|until| until.map(|u| u - now)); - Self::save_locked(st, left, quiet_left) + self.save_locked(st, left, quiet_left) }) } fn load_state(&self, v: &toml::Value) -> Result<(), String> { self.quiesced(|st| { let now = self.now_ns(st); - Self::load_locked(st, v, now); + self.load_locked(st, v, now); + self.cancel.store(0, Ordering::Release); self.publish_gen(st); }); self.sync_irq(); @@ -2057,7 +2279,7 @@ impl crate::traits::Saveable for Isp1040 { } impl Isp1040 { - fn save_locked(st: &State, selecting_left: Option, quiet_left: [Option; BUSES]) -> toml::Value { + fn save_locked(&self, st: &State, selecting_left: Option, quiet_left: [Option; BUSES]) -> toml::Value { let mut t = toml::map::Map::new(); for (k, v) in [ ("bus_cfg1", st.bus_cfg1), ("bus_ctrl", st.bus_ctrl), ("bus_status", st.bus_status), @@ -2130,16 +2352,18 @@ impl Isp1040 { m.insert("sense".into(), u8_slice_to_toml(&sel.done.sense)); t.insert("selecting".into(), toml::Value::Table(m)); } + self.part.save_extra(st, &mut t); toml::Value::Table(t) } /// Unprocessed request entries survive: the ring thread is kicked and /// runs them after the load. - fn load_locked(st: &mut State, v: &toml::Value, now: i64) { + fn load_locked(&self, st: &mut State, v: &toml::Value, now: i64) { { let ctl = std::mem::take(&mut st.ctl); *st = State::new(); st.ctl = ctl; + self.part.load_extra(st, v); st.ctl.gen += 1; st.ctl.rqst_epoch += 1; st.ctl.resp_epoch += 1; diff --git a/src/dev/isp1040/tests.rs b/src/dev/isp1040/tests.rs index f911743fd..d4f701970 100644 --- a/src/dev/isp1040/tests.rs +++ b/src/dev/isp1040/tests.rs @@ -2,7 +2,8 @@ //! and ring contract. Host memory is a flat byte array behind a fake `PciDma`; the disk //! is a small raw image. //! -//! Ported from tests/qtest/qlogic-isp-test.c in jamesbraid/qemu-sgi-origami. +//! Ported from tests/qtest/qlogic-isp-test.c and tests/qtest/sn0-isp12160-test.c +//! in jamesbraid/qemu-sgi-origami. use super::*; use crate::irq::LevelProbe; @@ -59,15 +60,16 @@ fn image(name: &str, blocks: usize) -> std::path::PathBuf { } impl Rig { - /// A chip on fake memory, no disks. - fn bare() -> Rig { + fn on(isp: Isp1040) -> Rig { let mem = FakeMem::new(); - let isp = Isp1040::new(); Device::start(&isp); - isp.connect_dma(mem.clone()); + isp.state().connect_dma(mem.clone()); Rig { isp, mem } } + /// A chip on fake memory, no disks. + fn bare() -> Rig { Rig::on(Isp1040::new()) } + /// A chip with a disk of `blocks` blocks at target `id`. fn with_disk(name: &str, id: u8, blocks: usize) -> Rig { let r = Rig::bare(); @@ -165,7 +167,7 @@ fn cdb10(op: u8, lba: u32, blocks: u16) -> [u8; 10] { #[test] fn identity_and_window() { let r = Rig::bare(); - let c = r.isp.config(); + let c = r.isp.state().config(); assert_eq!(c.read(0), 0x1020_1077, "device and vendor id"); assert_eq!(c.read(2) & 0xff, 0x05, "revision"); assert_eq!(c.read(2) >> 16, 0x0100, "mass storage, SCSI"); @@ -239,7 +241,8 @@ fn dma_block_config_is_storage_and_control_is_inert() { fn bus_ctrl_status_and_the_interrupt_line() { let r = Rig::bare(); let (probe, line) = LevelProbe::line(); - r.isp.connect_interrupt(line); + r.isp.state().connect_irq(line); + r.isp.attached(); r.w(reg::BUS_CTRL, CTRL_INT_ENABLE | CTRL_RISC_INT_ENABLE); assert_eq!(r.r(reg::BUS_CTRL), CTRL_INT_ENABLE | CTRL_RISC_INT_ENABLE); r.w(reg::BUS_CTRL, 0xffff); @@ -1048,7 +1051,8 @@ fn an_acknowledgement_does_not_erase_a_later_completion() { fn clearing_the_interrupt_keeps_a_completion_the_driver_was_not_shown() { let r = Rig::with_disk("unseen", 0, 64); let (probe, line) = LevelProbe::line(); - r.isp.connect_interrupt(line); + r.isp.state().connect_irq(line); + r.isp.attached(); r.init_rings(8, 8); r.w(reg::BUS_CTRL, CTRL_INT_ENABLE | CTRL_RISC_INT_ENABLE); r.put_entry(0, &tur_entry(0x5001)); @@ -1248,6 +1252,30 @@ fn save_load_round_trip() { assert_eq!(dst.r(reg::REQUEST_QUEUE), 4, "the held completion and the ring carry over"); } +/// A restore into a chip that has been in use equals one into a fresh chip: +/// the snapshot names everything the chip holds, and nothing of the old +/// rings, held completions or mailboxes survives the load. +#[test] +fn isp1040_load_into_a_dirtied_instance_equals_load_into_a_fresh_one() { + let src = Rig::with_disk("ld-src", 0, 64); + src.init_rings(8, 3); + src.cmd(&[mbox::WRITE_RAM_WORD, 0x40, 0x1234]); + for i in 0..4 { src.put_entry(i, &tur_entry(0x5000 + i)); } + src.doorbell(4); + let v = src.isp.save_state(); + + let used = Rig::with_disk("ld-used", 0, 64); + used.init_rings(8, 5); + used.cmd(&[mbox::WRITE_RAM_WORD, 0x41, 0x9999]); + for i in 0..6 { used.put_entry(i, &tur_entry(0x6000 + i)); } + used.doorbell(6); + let fresh = Rig::with_disk("ld-fresh", 0, 64); + used.isp.load_state(&v).unwrap(); + fresh.isp.load_state(&v).unwrap(); + assert_eq!(used.isp.save_state(), fresh.isp.save_state()); + assert_eq!(used.isp.save_state(), v); +} + #[test] fn a_disk_attaches_where_asked_and_the_monitor_can_see_it() { use crate::scsi_monitor::ScsiTargets; @@ -1293,7 +1321,7 @@ fn dma_above_4_gib_reaches_the_bus_in_full() { let isp = Isp1040::new(); Device::start(&isp); let probe = Arc::new(AddrProbe(Mutex::new(Vec::new()))); - isp.connect_dma(probe.clone()); + isp.state().connect_dma(probe.clone()); let mut b = [0u8; 4]; assert!(isp.dma_read(0x1_2345_6000, &mut b)); assert!(isp.dma_write(0xffff_ffff_ffff_f000, &b)); @@ -1355,10 +1383,11 @@ fn gated_rig(name: &str) -> (Rig, Arc, Arc) { let gate = GatedMem::new(mem.clone(), BUF); let isp = Isp1040::new(); Device::start(&isp); - isp.connect_dma(gate.clone()); + isp.state().connect_dma(gate.clone()); isp.bus().attach_image(1, &image(name, 64), false).expect("attach"); let (probe, line) = LevelProbe::line(); - isp.connect_interrupt(line); + isp.state().connect_irq(line); + isp.attached(); let r = Rig { isp, mem }; r.init_rings(4, 4); r.w(reg::BUS_CTRL, CTRL_INT_ENABLE | CTRL_RISC_INT_ENABLE); @@ -1395,11 +1424,12 @@ fn a_soft_reset_does_not_wait_for_the_target() { let gate = GatedMem::new(mem.clone(), BUF); let isp = Isp1040::new(); Device::start(&isp); - isp.connect_dma(gate.clone()); + isp.state().connect_dma(gate.clone()); isp.bus().attach_image(1, &image("async-reset", 64), false).expect("attach"); let r = Rig { isp, mem }; let (probe, line) = LevelProbe::line(); - r.isp.connect_interrupt(line); + r.isp.state().connect_irq(line); + r.isp.attached(); r.init_rings(4, 4); r.w(reg::BUS_CTRL, CTRL_INT_ENABLE | CTRL_RISC_INT_ENABLE); r.mem.put(BUF, &[0x5a; BLOCK_SIZE]); @@ -1574,7 +1604,7 @@ fn the_ring_thread_lives_between_start_and_stop() { let isp = Isp1040::new(); assert!(!isp.is_running(), "the constructor starts no thread"); let mem = FakeMem::new(); - isp.connect_dma(mem.clone()); + isp.state().connect_dma(mem.clone()); isp.bus().attach_image(0, &image("lifecycle", 64), false).expect("attach"); let r = Rig { isp, mem }; r.init_rings(8, 8); @@ -1618,6 +1648,403 @@ fn a_controller_built_on_a_filled_bus_answers_for_its_target() { let _ = std::fs::remove_file(img); } + +struct MarkerTarget(u8); + +impl crate::scsi::ScsiTarget for MarkerTarget { + fn request(&mut self, _req: &crate::scsi::ScsiRequest) -> std::io::Result { + Ok(crate::scsi::ScsiResponse { status: 0, data: vec![self.0] }) + } +} + +/// Target 0 on channel 1 must stay distinct from target 0 on channel 0. +#[test] +fn the_isp12160_routes_ring_requests_to_separate_buses() { + let bus0 = Isp1040::new_bus(); + let bus1 = Isp1040::new_bus(); + bus0.attach(0, Box::new(MarkerTarget(0x10))).unwrap(); + bus1.attach(0, Box::new(MarkerTarget(0x20))).unwrap(); + let r = Rig::on(Isp1040::new_12160(bus0, bus1)); + r.init_rings(4, 4); + + r.put_entry(0, &request(0x1234, 0x80, &[0x12, 0, 0, 0, 36, 0], &[(BUF, 36)])); + r.doorbell(1); + assert_eq!(r.mem.get(BUF, 1), vec![0x20], "channel 1 reaches its target 0"); + + r.put_entry(1, &request(0x1235, 0, &[0x12, 0, 0, 0, 36, 0], &[(BUF + 64, 36)])); + r.doorbell(2); + assert_eq!(r.mem.get(BUF + 64, 1), vec![0x10], "channel 0 keeps its own target 0"); +} + +struct ResetProbe(Arc); + +impl crate::scsi::ScsiTarget for ResetProbe { + fn request(&mut self, _req: &crate::scsi::ScsiRequest) -> std::io::Result { + Ok(crate::scsi::ScsiResponse { status: 0, data: Vec::new() }) + } + + fn bus_reset(&mut self) { + self.0.fetch_add(1, Ordering::Relaxed); + } +} + +/// A snapshot retains both the chip-wide reset and the selected target reset, +/// and the resumed worker delivers both through the original channel buses. +#[test] +fn the_isp12160_preserves_queued_bus_and_target_resets_across_save_load() { + let src = Isp1040::new_12160(Isp1040::new_bus(), Isp1040::new_bus()); + { + let mut state = src.st.lock(); + state.reset_targets = true; + state.pending_target_resets.push_back((1, Some(2))); + } + let saved = src.save_state(); + + let resets0 = Arc::new(std::sync::atomic::AtomicUsize::new(0)); + let resets1 = Arc::new(std::sync::atomic::AtomicUsize::new(0)); + let selected_reset = Arc::new(std::sync::atomic::AtomicUsize::new(0)); + let bus0 = Isp1040::new_bus(); + let bus1 = Isp1040::new_bus(); + bus0.attach(1, Box::new(ResetProbe(resets0.clone()))).unwrap(); + bus1.attach(1, Box::new(ResetProbe(resets1.clone()))).unwrap(); + bus1.attach(2, Box::new(ResetProbe(selected_reset.clone()))).unwrap(); + let dst = Isp1040::new_12160(bus0, bus1); + dst.load_state(&saved).unwrap(); + Device::start(&dst); + dst.wait_idle(); + Device::stop(&dst); + + assert_eq!(resets0.load(Ordering::Relaxed), 1); + assert_eq!(resets1.load(Ordering::Relaxed), 1); + assert_eq!(selected_reset.load(Ordering::Relaxed), 2, "the full reset and queued target reset both run"); + assert_eq!(saved.get("reset_targets").and_then(toml::Value::as_bool), Some(true)); + assert_eq!(saved.get("target_resets").and_then(toml::Value::as_array).map(Vec::len), Some(1)); + let after = dst.save_state(); + assert_eq!(after.get("reset_targets").and_then(toml::Value::as_bool), Some(false)); + assert_eq!(after.get("target_resets").and_then(toml::Value::as_array).map(Vec::len), Some(0)); +} + +// ---- ISP12160 second channel and recovery commands ----------------------- + +fn rig12160() -> Rig { Rig::on(Isp1040::new_12160(Isp1040::new_bus(), Isp1040::new_bus())) } + +/// A 64-block image whose first block is `fill`, so a read says which disk answered. +fn filled_image(name: &str, fill: u8) -> std::path::PathBuf { + let p = image(name, 64); + let mut data = vec![0u8; 64 * BLOCK_SIZE]; + data[..BLOCK_SIZE].fill(fill); + std::fs::write(&p, data).expect("test image"); + p +} + +/// A 12160 with a different disk at target 1 on each channel. +fn disks_on_both_channels(name: &str) -> Rig { + let r = rig12160(); + for channel in 0..2u8 { + let img = filled_image(&format!("{name}{channel}"), 0xa0 + channel); + r.isp.bus_for_channel(channel).unwrap().attach_image(1, &img, false).expect("attach"); + } + r +} + +/// Issues entries one at a time on a 16-entry ring and returns each completion. +struct Seq<'a> { + r: &'a Rig, + n: u16, +} + +impl<'a> Seq<'a> { + fn new(r: &'a Rig) -> Self { + r.init_rings(16, 16); + Seq { r, n: 0 } + } + fn run(&mut self, handle: u32, target: u8, cdb: &[u8], segs: &[(u32, u32)]) -> [u8; ENTRY_SIZE] { + let slot = self.n; + self.r.put_entry(slot as u32, &request(handle, target, cdb, segs)); + self.n += 1; + self.r.doorbell(self.n); + self.r.status(slot as u32) + } +} + +const TUR: [u8; 6] = [0; 6]; + +/// Counts bus resets and answers any command with a full block pair, so a read +/// whose second buffer segment is unreachable stays outstanding. +struct CountingDisk(Arc); + +impl crate::scsi::ScsiTarget for CountingDisk { + fn request(&mut self, _req: &crate::scsi::ScsiRequest) -> std::io::Result { + Ok(crate::scsi::ScsiResponse { status: 0, data: vec![0; 2 * BLOCK_SIZE] }) + } + fn bus_reset(&mut self) { self.0.fetch_add(1, Ordering::Relaxed); } +} + +struct Held { + r: Rig, + /// Bus resets seen by the counting disks at target 1 on channel 0, target 1 on channel 1, + /// and target 2 on channel 1. + ch0: Arc, + ch1: Arc, + ch1_other: Arc, +} + +impl Held { + fn new() -> Held { + let counters: Vec<_> = (0..3).map(|_| Arc::new(std::sync::atomic::AtomicUsize::new(0))).collect(); + let (bus0, bus1) = (Isp1040::new_bus(), Isp1040::new_bus()); + bus0.attach(1, Box::new(CountingDisk(counters[0].clone()))).unwrap(); + bus1.attach(1, Box::new(CountingDisk(counters[1].clone()))).unwrap(); + bus1.attach(2, Box::new(CountingDisk(counters[2].clone()))).unwrap(); + let r = Rig::on(Isp1040::new_12160(bus0, bus1)); + r.init_rings(16, 16); + Held { r, ch0: counters[0].clone(), ch1: counters[1].clone(), ch1_other: counters[2].clone() } + } + /// A read whose payload store the fabric refuses: the adapter keeps it outstanding, which + /// is what the abort and reset commands act on. + fn hold(&self, slot: u32, handle: u32, target: u8) { + self.r.put_entry(slot, &request(handle, target, &cdb10(scsi_cmd::READ_10, 0, 2), + &[(BUF, BLOCK_SIZE as u32), (MEM as u32, BLOCK_SIZE as u32)])); + self.r.doorbell(slot as u16 + 1); + assert_eq!(g32(&self.r.status(slot), 4), 0, "handle {handle:#x} is held, not completed"); + } + fn command(&self, regs: &[u16]) -> u16 { + let status = self.r.cmd(regs); + self.r.isp.wait_idle(); + status + } + /// The completion posted in response slot `slot`: (handle, completion code). + fn answer(&self, slot: u32) -> (u32, u16) { + let s = self.r.status(slot); + (g32(&s, 4), g16(&s, 10)) + } +} + +const CH1: u16 = 0x8000; + +/// A driver sizing its queues and recognising the part reads these before any ring exists. +#[test] +fn the_isp12160_reports_its_own_identity_firmware_and_unmapped_registers() { + let r = rig12160(); + assert_eq!(r.r(reg::ID_HIGH), 0x1216); + assert_eq!(r.r(reg::BUS_CFG0), 0, "no revision field on this part"); + r.w(reg::DMA_CMD_CFG, 0xffff); + r.w(reg::DMA_DATA_CFG, 0xffff); + assert_eq!((r.r(reg::DMA_CMD_CFG), r.r(reg::DMA_DATA_CFG)), (0, 0), "the flat ISP1040 DMA blocks are not here"); + + // Mailbox 6 is seeded because an untouched output would also read zero. + r.w(r.mb(6), 0x1234); + r.w(r.mb(0), mbox::ABOUT_FIRMWARE); + r.w(reg::HOST_CMD, 0x5 << HOST_CMD_SHIFT); + assert_eq!(r.r(r.mb(0)), mbox::COMPLETE); + assert_eq!([r.r(r.mb(1)), r.r(r.mb(2))], [10, 4]); + assert_eq!(r.r(r.mb(6)), 0, "the attribute word is cleared, not echoed"); + r.ack(); + r.w(r.mb(0), mbox::GET_FIRMWARE_STATUS); + r.w(reg::HOST_CMD, 0x5 << HOST_CMD_SHIFT); + assert_eq!(r.r(r.mb(0)), mbox::COMPLETE); + assert_eq!([r.r(r.mb(1)), r.r(r.mb(2))], [0, MAX_REQUESTS]); + r.ack(); +} + +/// The driver switches register banks through bus configuration 1 bits 9:8; each of the +/// four selections must show its own words, and no other bit may stick. +#[test] +fn the_isp12160_selects_four_distinct_banks_and_keeps_only_the_bank_field() { + let r = rig12160(); + let selections = [0x0000, 0x0100, 0x0200, 0x0300]; + for sel in selections { + r.w(reg::BUS_CFG1, sel); + r.w(reg::BANK_BASE, 0x1000 + sel); + r.w(reg::BANK_BASE + 0x20, 0x2000 + sel); + } + for sel in selections { + r.w(reg::BUS_CFG1, sel); + assert_eq!(r.r(reg::BANK_BASE), 0x1000 + sel, "bank {sel:#x}"); + assert_eq!(r.r(reg::BANK_BASE + 0x20), 0x2000 + sel, "bank {sel:#x}"); + } + r.w(reg::BUS_CFG1, 0xb2c5); + assert_eq!(r.r(reg::BUS_CFG1), 0x0200); + + // A soft reset hands a re-initialising driver empty banks, not the previous run's words. + r.w(reg::BUS_CTRL, CTRL_SOFT_RESET); + assert_eq!(r.r(reg::BUS_CFG1), 0); + for sel in selections { + r.w(reg::BUS_CFG1, sel); + assert_eq!(r.r(reg::BANK_BASE), 0, "bank {sel:#x} after reset"); + } +} + +/// A second command submitted before the driver acknowledged the first result must not +/// run, or it would overwrite the answer the driver has not read. +#[test] +fn the_isp12160_ignores_a_mailbox_submission_while_the_result_is_unacknowledged() { + let r = rig12160(); + r.w(r.mb(0), mbox::NOP); + r.w(reg::HOST_CMD, 0x5 << HOST_CMD_SHIFT); + assert_eq!(r.r(r.mb(0)), mbox::COMPLETE); + assert_eq!(r.r(reg::SEMAPHORE) & SEMAPHORE_LOCK, SEMAPHORE_LOCK); + + r.w(r.mb(1), 0x9999); + r.w(r.mb(0), mbox::ABOUT_FIRMWARE); + r.w(reg::HOST_CMD, 0x5 << HOST_CMD_SHIFT); + assert_eq!(r.r(r.mb(0)), mbox::ABOUT_FIRMWARE, "the second submission did not run"); + assert_eq!(r.r(r.mb(1)), 0x9999); + + r.ack(); + r.w(reg::HOST_CMD, 0x5 << HOST_CMD_SHIFT); + assert_eq!(r.r(r.mb(0)), mbox::COMPLETE); + assert_eq!(r.r(r.mb(1)), 10, "acknowledged, the same submission runs"); + r.ack(); +} + +/// An empty slot on either channel times out on its own bus; a driver probing channel 1 +/// must see an absent target, not a hang or the channel 0 answer. +#[test] +fn a_missing_target_times_out_on_each_channel() { + let r = rig12160(); + r.init_rings(16, 16); + r.clock(0); + // An absent target answers once its selection timeout has run out on + // the adapter's clock. + let mut now = 0; + for (slot, (handle, target)) in [(0x31, 5), (0x32, 0x80 | 5)].into_iter().enumerate() { + r.put_entry(slot as u32, &request(handle, target, &TUR, &[])); + r.doorbell(slot as u16 + 1); + assert_ne!(g32(&r.status(slot as u32), 4), handle, "nothing before the selection timeout"); + now += DEFAULT_SELECTION_TIMEOUT as i64 * MS; + r.clock(now); + let s = r.status(slot as u32); + assert_eq!(g32(&s, 4), handle); + assert_eq!(g16(&s, 10), IOCB_INCOMPLETE); + assert_eq!(g16(&s, 12), STATE_GOT_BUS); + assert_eq!(g16(&s, 14), STATUS_FLAG_TIMEOUT); + } +} + +/// A command refused on channel 1 must not disturb its neighbour on channel 0, and +/// a write must land on the disk of the channel it named. +#[test] +fn a_refused_command_on_channel_1_leaves_channel_0_and_the_media_alone() { + let r = disks_on_both_channels("refuse"); + let mut seq = Seq::new(&r); + // Collect each disk's power-on unit attention so the reads below see real answers. + seq.run(0xa0, 1, &TUR, &[]); + seq.run(0xa1, 0x81, &TUR, &[]); + + r.mem.put(BUF, &[0x5c; BLOCK_SIZE]); + let beyond_disk = seq.run(0xb1, 0x81, &cdb10(scsi_cmd::READ_10, 1000, 1), &[(BUF, BLOCK_SIZE as u32)]); + assert_eq!(g16(&beyond_disk, 8) & 0xff, CHECK_CONDITION as u16); + assert_eq!(r.mem.get(BUF, BLOCK_SIZE), vec![0x5c; BLOCK_SIZE], "the refused read stored nothing"); + + let good = seq.run(0xb0, 1, &cdb10(scsi_cmd::READ_10, 0, 1), &[(BUF + 0x1000, BLOCK_SIZE as u32)]); + assert_eq!(g16(&good, 10), IOCB_COMPLETE); + assert_eq!(r.mem.get(BUF + 0x1000, BLOCK_SIZE), vec![0xa0; BLOCK_SIZE], "channel 0 still reads its own disk"); + + r.mem.put(BUF + 0x2000, &[0x77; BLOCK_SIZE]); + seq.run(0xc0, 0x81, &cdb10(scsi_cmd::WRITE_10, 3, 1), &[(BUF + 0x2000, BLOCK_SIZE as u32)]); + seq.run(0xc1, 0x81, &cdb10(scsi_cmd::READ_10, 3, 1), &[(BUF + 0x3000, BLOCK_SIZE as u32)]); + seq.run(0xc2, 1, &cdb10(scsi_cmd::READ_10, 3, 1), &[(BUF + 0x4000, BLOCK_SIZE as u32)]); + assert_eq!(r.mem.get(BUF + 0x3000, BLOCK_SIZE), vec![0x77; BLOCK_SIZE], "channel 1 holds what was written"); + assert_eq!(r.mem.get(BUF + 0x4000, BLOCK_SIZE), vec![0; BLOCK_SIZE], "channel 0's block 3 is untouched"); +} + +/// Abort device names a channel in the top bit of its selector; the same target number +/// held on the other bus must stay outstanding. +#[test] +fn abort_device_retires_only_the_named_channels_commands() { + let h = Held::new(); + h.hold(0, 0x41, 1); + h.hold(1, 0x42, 0x81); + assert_eq!(h.command(&[mbox::ABORT_DEVICE, CH1 | 0x0100, 0, 0]), mbox::COMPLETE); + assert_eq!(h.answer(0), (0x42, IOCB_ABORTED)); + assert_eq!(h.answer(1), (0, 0), "channel 0's command is still held"); + // The survivor is still the adapter's to abort. + assert_eq!(h.command(&[mbox::ABORT_DEVICE, 0x0100, 0, 0]), mbox::COMPLETE); + assert_eq!(h.answer(1), (0x41, IOCB_ABORTED)); +} + +/// Abort command names one handle: two commands to the same channel, target and unit differ +/// only by it, and a selector that matches a held handle on the wrong channel is an error. +#[test] +fn abort_command_matches_the_handle_and_the_whole_selector() { + let h = Held::new(); + h.hold(0, 0x51, 0x81); + h.hold(1, 0x52, 0x81); + assert_eq!(h.command(&[mbox::ABORT_COMMAND, CH1 | 0x0100, 0, 0x52]), mbox::COMPLETE); + assert_eq!(h.answer(0), (0x52, IOCB_ABORTED)); + assert_eq!(h.answer(1), (0, 0), "its neighbour on the same target is untouched"); + + assert_eq!(h.command(&[mbox::ABORT_COMMAND, 0x0100, 0, 0x51]), mbox::COMMAND_ERROR, + "a held handle named on the wrong channel is not a match"); + assert_eq!(h.answer(1), (0, 0)); + assert_eq!(h.command(&[mbox::ABORT_COMMAND, CH1 | 0x0100, 0, 0x51]), mbox::COMPLETE); + assert_eq!(h.answer(1), (0x51, IOCB_ABORTED)); +} + +/// A driver clearing a device it already drained must read success, while naming a handle +/// nothing is using must read as an error rather than claim a retirement. +#[test] +fn aborting_nothing_is_complete_but_aborting_an_unknown_handle_is_refused() { + let h = Held::new(); + assert_eq!(h.command(&[mbox::ABORT_DEVICE, CH1 | 0x0100, 0, 0]), mbox::COMPLETE); + assert_eq!(h.command(&[mbox::ABORT_COMMAND, CH1 | 0x0100, 0, 0x77]), mbox::COMMAND_ERROR); + assert_eq!(h.answer(0), (0, 0), "neither posted a completion"); +} + +/// Target reset reaches one target on one channel; the same number on the other channel and +/// the neighbour on the same channel must not see it, and the reset target keeps serving. +#[test] +fn target_reset_is_scoped_to_one_target_on_one_channel() { + let h = Held::new(); + h.hold(0, 0x41, 1); + h.hold(1, 0x42, 0x81); + assert_eq!(h.command(&[mbox::ABORT_TARGET, CH1 | 0x0100, 0, 0]), mbox::COMPLETE); + assert_eq!(h.answer(0), (0x42, IOCB_RESET)); + assert_eq!(h.answer(1), (0, 0)); + assert_eq!([h.ch0.load(Ordering::Relaxed), h.ch1.load(Ordering::Relaxed), h.ch1_other.load(Ordering::Relaxed)], + [0, 1, 0]); + + h.r.put_entry(2, &request(0x43, 0x81, &TUR, &[])); + h.r.doorbell(3); + assert_eq!(h.answer(1), (0x43, IOCB_COMPLETE)); +} + +/// Bus reset takes its channel from mailbox 2; mailbox 1 is the reset delay and here holds a value +/// that is not even a channel. The whole named bus resets, the other one does not. +#[test] +fn bus_reset_takes_its_channel_from_mailbox_two_and_resets_every_target_on_it() { + let h = Held::new(); + h.hold(0, 0x41, 1); + h.hold(1, 0x42, 0x81); + assert_eq!(h.command(&[mbox::BUS_RESET, 3, 1]), mbox::COMPLETE); + assert_eq!(h.answer(0), (0x42, IOCB_RESET)); + assert_eq!(h.answer(1), (0, 0)); + assert_eq!([h.ch0.load(Ordering::Relaxed), h.ch1.load(Ordering::Relaxed), h.ch1_other.load(Ordering::Relaxed)], + [0, 1, 1]); + assert_eq!(h.command(&[mbox::BUS_RESET, 0, 2]), mbox::PARAM_ERROR, "there is no third bus"); +} + +/// A bus reset leaves a unit attention on the disks of that channel only; the driver's +/// first command afterwards is expected to collect it. +#[test] +fn a_bus_reset_leaves_a_unit_attention_on_the_reset_channel_only() { + let r = disks_on_both_channels("uareset"); + let mut seq = Seq::new(&r); + seq.run(0x10, 1, &TUR, &[]); + seq.run(0x11, 0x81, &TUR, &[]); + assert_eq!(g16(&seq.run(0x12, 0x81, &TUR, &[]), 8) & 0xff, 0, "attention was collected"); + + assert_eq!(r.cmd(&[mbox::BUS_RESET, 0, 1]), mbox::COMPLETE); + r.isp.wait_idle(); + let after = seq.run(0x13, 0x81, &TUR, &[]); + assert_eq!(g16(&after, 8) & 0xff, CHECK_CONDITION as u16); + assert_eq!(after[34] & 0x0f, 0x06, "unit attention"); + let other = seq.run(0x14, 1, &TUR, &[]); + assert_eq!(g16(&other, 8) & 0xff, 0, "channel 0 was not reset"); + assert_eq!(g16(&seq.run(0x15, 0x81, &TUR, &[]), 8) & 0xff, 0, "reported once"); +} + // ---- the hold-off after a bus reset --------------------------------------- // // Datasheet, "Bus-reset hold-off (0x0018)": after the reset the chip starts no @@ -1653,6 +2080,87 @@ fn a_bus_reset_puts_a_disks_write_cache_back() { assert_eq!(r.mem.get(BUF + 6, 1)[0] & 0x04, 0x04, "WCE is back"); } +/// A 12160 with a cached disk at target 1 on each channel whose guests cleared WCE on both. +fn cleared_write_cache_on_both_channels(name: &str) -> Rig { + let r = rig12160(); + for channel in 0..2u8 { + let bus = r.isp.bus_for_channel(channel).unwrap(); + bus.attach_cached_image(1, &image(&format!("{name}{channel}"), 64)).expect("attach"); + bus.select_write_cache(1, false); + assert!(!bus.write_cache_reported(1), "channel {channel}: the guest cleared it"); + } + r +} + +/// The write cache bit MODE SENSE(6) of the caching page reports for target 1 on `channel`. A TUR +/// goes first: it collects the reset's unit attention, and consumes the reset flag as a guest's +/// first command does, so the mode sense that follows returns its page. +fn wce_after_next_command(seq: &mut Seq, channel: u8, handle: u32) -> bool { + let target = channel << 7 | 1; + seq.run(handle, target, &TUR, &[]); + let done = seq.run(handle + 0x100, target, &[0x1a, 0x08, 0x08, 0, 24, 0], &[(BUF, 24)]); + assert_eq!(g16(&done, 10), IOCB_COMPLETE); + assert_eq!(seq.r.mem.get(BUF + 4, 1)[0], 0x08, "caching page"); + seq.r.mem.get(BUF + 6, 1)[0] & 0x04 != 0 +} + +/// Each channel's disks put their write cache back on that channel's reset and not on the +/// other's, so the flag has to be set for the channel the mailbox names. +#[test] +fn a_12160_bus_reset_of_channel_1_restores_write_cache_on_channel_1_only() { + let r = cleared_write_cache_on_both_channels("wce12160-ch1"); + let mut seq = Seq::new(&r); + assert_eq!(r.cmd(&[mbox::BUS_RESET, 0, 1]), mbox::COMPLETE); + r.isp.wait_idle(); + assert!(!wce_after_next_command(&mut seq, 0, 0x31), "channel 0 was not reset"); + assert!(wce_after_next_command(&mut seq, 1, 0x32), "channel 1 was reset"); +} + +#[test] +fn a_12160_bus_reset_of_channel_0_restores_write_cache_on_channel_0_only() { + let r = cleared_write_cache_on_both_channels("wce12160-ch0"); + let mut seq = Seq::new(&r); + assert_eq!(r.cmd(&[mbox::BUS_RESET, 0, 0]), mbox::COMPLETE); + r.isp.wait_idle(); + assert!(!wce_after_next_command(&mut seq, 1, 0x33), "channel 1 was not reset"); + assert!(wce_after_next_command(&mut seq, 0, 0x34), "channel 0 was reset"); +} + +/// Both reset the chip's SCSI buses, which the disks see as a bus reset on each channel. +#[test] +fn a_12160_chip_reset_restores_write_cache_on_both_channels() { + let r = cleared_write_cache_on_both_channels("wce12160-chip"); + r.isp.reset(); + r.isp.wait_idle(); + let mut seq = Seq::new(&r); + assert!(wce_after_next_command(&mut seq, 0, 0x35), "channel 0"); + assert!(wce_after_next_command(&mut seq, 1, 0x36), "channel 1"); +} + +#[test] +fn a_12160_soft_reset_restores_write_cache_on_both_channels() { + let r = cleared_write_cache_on_both_channels("wce12160-soft"); + r.w(reg::BUS_CTRL, CTRL_SOFT_RESET); + r.isp.wait_idle(); + let mut seq = Seq::new(&r); + assert!(wce_after_next_command(&mut seq, 0, 0x37), "channel 0"); + assert!(wce_after_next_command(&mut seq, 1, 0x38), "channel 1"); +} + +/// The single-bus part has no bus 1: a mailbox that names it is ignored, and the one bus it has +/// keeps its disks' write cache as the guest set it. +#[test] +fn an_isp1040_bus_reset_naming_bus_1_leaves_the_reset_flag_unset() { + let r = Rig::bare(); + r.isp.bus().attach_cached_image(1, &image("wce-bus1", 64)).expect("attach"); + r.init_rings(4, 4); + r.isp.bus().select_write_cache(1, false); + bus_reset(&r, 0, 1); + r.isp.wait_idle(); + assert!(!r.isp.targets_reset[0].load(Ordering::Acquire)); + assert!(!r.isp.targets_reset[1].load(Ordering::Acquire)); +} + /// A TUR for the disk at target 1. fn tur1(handle: u32) -> [u8; ENTRY_SIZE] { request(handle, 1, &[0, 0, 0, 0, 0, 0], &[]) diff --git a/src/dev/isp12160.rs b/src/dev/isp12160.rs new file mode 100644 index 000000000..e0cf902de --- /dev/null +++ b/src/dev/isp12160.rs @@ -0,0 +1,337 @@ +//! The QLogic ISP12160: two SCSI buses behind one PCI function. +//! +//! The ISP12160 shares the ISP1040 ring, DMA, and mailbox mechanism, while +//! selecting targets from either channel. Each channel remains an explicit +//! [`ScsiBus`], including for monitor and run-time disc operations. +//! +//! Ported from hw/scsi/qlogic-isp.c and include/hw/scsi/qlogic-isp.h +//! in jamesbraid/qemu-sgi-origami. + +use std::collections::VecDeque; +use std::ops::Deref; + +use parking_lot::MutexGuard; + +use crate::dev::isp1040::{ + mbox, reg, Bank, Core, Isp1040, IspPart, State, CFG1_BANK_SELECT, FW_VERSION, IOCB_ABORTED, IOCB_RESET, MAILBOX_COUNT, + MAX_REQUESTS, MAX_TARGET, PCI_CLASS_SCSI, PCI_VENDOR_QLOGIC, SEMAPHORE_LOCK, STATE_ALL_PHASES, +}; +use crate::irq::IrqLine; +use crate::pci::{Bar, PciConfig, PciConfigPolicy, PciFunction, PciFunctionState, PciRegisterPolicy, PCI_INTA}; +use crate::scsi::{ScsiBus, ScsiIdentities, ScsiIdentity}; +use crate::snapshot::{get_field, hex_u16, hex_u32, load_u16_slice, toml_u16, toml_u32, toml_u8, u16_slice_to_toml}; +use crate::traits::{Device, Resettable, Saveable}; +use crate::workstation::DiscChanger; + +const ISP_DISK: ScsiIdentity = ScsiIdentity::new(b"SGI ", b"IRIS EMULATED ", b"1.0 "); +const ISP_CDROM: ScsiIdentity = ISP_DISK; + +pub(crate) struct Isp12160Part; + +impl IspPart for Isp12160Part { + fn name(&self) -> &'static str { "ISP12160" } + fn channels(&self) -> usize { 2 } + fn pci_config(&self, bars: [Bar; 6]) -> PciConfig { + let registers = [ + PciRegisterPolicy { reg_dword: 13, initial: 0x44, writable_mask: 0 }, + PciRegisterPolicy { reg_dword: 17, initial: 0x0001_0001, writable_mask: 0 }, + PciRegisterPolicy { reg_dword: 18, initial: 0, writable_mask: 0x100 }, + ]; + PciConfig::new_with_policy( + PCI_VENDOR_QLOGIC, + 0x1216, + PCI_CLASS_SCSI, + 6, + bars, + PCI_INTA, + PciConfigPolicy { + status: 0x02b0, + status_w1c_mask: 0, + command_mask: 0x0156, + latency_mask: 0xff, + cache_line_mask: 0xff, + registers: ®isters, + }, + ) + } + fn firmware_version(&self) -> [u16; 3] { [10, 4, FW_VERSION[2]] } + fn active_bank(&self, cfg1: u16) -> Bank { + match (cfg1 >> 8) & 3 { + 1 => Bank::Sxp, + 2 => Bank::Sxp1, + 3 => Bank::Dma, + _ => Bank::Risc, + } + } + fn checks_ram_bounds(&self) -> bool { true } + fn has_abort_commands(&self) -> bool { true } + fn target_on(&self, channel: u8, target: u8) -> u8 { target | (channel << 4) } + fn decode_target(&self, target: u8) -> (usize, u8) { ((target >> 4) as usize, target & 0x0f) } + fn id_high(&self) -> u16 { 0x1216 } + fn bus_cfg0(&self) -> u16 { 0 } + fn cfg1_write_mask(&self) -> u16 { 0x0300 } + fn unmodelled_dma_register(&self, off: u32) -> bool { + matches!(off, reg::DMA_CMD_CFG | reg::DMA_CMD_CTRL | reg::DMA_DATA_CFG | reg::DMA_DATA_CTRL) + } + fn completion_phases(&self, moved: u64, total: u64) -> u16 { + let mut flags = 0x1700; + if moved != 0 { + flags |= 0x0800; + if moved == total { flags |= 0x4000; } + } + flags + } + fn mailbox_ready(&self, semaphore: u16) -> bool { semaphore & SEMAPHORE_LOCK == 0 } + fn clear_target_parameters(&self, mailbox: &mut [u16; MAILBOX_COUNT]) { mailbox[6] = 0; } + fn set_reset_delay(&self, st: &mut State) { + st.reset_delay = st.mailbox[1]; + st.reset_delay1 = st.mailbox[2]; + } + fn reset_targets(&self, st: &mut State) -> bool { + st.reset_targets = true; + true + } + fn execute_mailbox(&self, core: &Core, st: &mut MutexGuard<'_, State>) -> bool { + let cmd = st.mailbox[0]; + match cmd { + mbox::LOAD_RAM_A64 => st.mailbox[0] = core.ram_transfer(st, true), + mbox::ABORT_COMMAND | mbox::ABORT_DEVICE | mbox::ABORT_TARGET => { + let device = st.mailbox[1]; + let valid_target = (device >> 8) & 0x7f <= MAX_TARGET as u16; + let target = ((device >> 8) & 0x7f) as u8 | (((device >> 15) as u8) << 4); + let lun = device as u8; + let handle = ((st.mailbox[2] as u32) << 16) | st.mailbox[3] as u32; + let hit = valid_target && st.active.is_some_and(|(t, l, h)| t == target + && (cmd == mbox::ABORT_TARGET || l == lun) && (cmd != mbox::ABORT_COMMAND || h == handle)); + let completion = if cmd == mbox::ABORT_TARGET { IOCB_RESET } else { IOCB_ABORTED }; + if hit { core.cancel_active(st, completion); } + let retired = core.retire_outstanding(st, |(t, l, h)| valid_target && t == target + && (cmd == mbox::ABORT_TARGET || l == lun) && (cmd != mbox::ABORT_COMMAND || h == handle), completion); + if cmd == mbox::ABORT_TARGET && valid_target { + st.pending_target_resets.push_back((target >> 4, Some(target & 15))); + } + st.mailbox[0] = if cmd == mbox::ABORT_COMMAND && !hit && retired == 0 { mbox::COMMAND_ERROR } else { mbox::COMPLETE }; + } + // Mailbox 1 is the reset delay; mailbox 2 selects the bus. + mbox::BUS_RESET => { + let channel = st.mailbox[2]; + if channel > 1 { st.mailbox[0] = mbox::PARAM_ERROR; } + else { + if st.active.is_some_and(|(target, _, _)| target >> 4 == channel as u8) { + core.cancel_active(st, IOCB_RESET); + } + core.retire_outstanding(st, |(target, _, _)| target >> 4 == channel as u8, IOCB_RESET); + st.pending_target_resets.push_back((channel as u8, None)); + core.note_bus_reset(usize::from(channel)); + core.kick(st); + st.mailbox[0] = mbox::COMPLETE; + } + } + _ => return false, + } + true + } + fn save_extra(&self, st: &State, t: &mut toml::map::Map) { + t.insert("reset_targets".into(), toml::Value::Boolean(st.reset_targets)); + t.insert("target_resets".into(), toml::Value::Array(st.pending_target_resets.iter().map(|(channel, target)| { + toml::Value::Array(vec![hex_u16(*channel as u16), hex_u16(target.unwrap_or(0xff) as u16)]) + }).collect())); + t.insert("sxp1_bank".into(), u16_slice_to_toml(&st.sxp1_bank)); + t.insert("dma_bank".into(), u16_slice_to_toml(&st.dma_bank)); + t.insert("reset_delay1".into(), hex_u16(st.reset_delay1)); + t.insert("outstanding".into(), toml::Value::Array(st.outstanding.iter().map(|&(target, lun, handle)| { + toml::Value::Array(vec![hex_u16(target as u16), hex_u16(lun as u16), hex_u32(handle)]) + }).collect())); + } + fn load_extra(&self, st: &mut State, v: &toml::Value) { + st.reset_targets = get_field(v, "reset_targets").and_then(|x| x.as_bool()).unwrap_or(false); + if let Some(toml::Value::Array(items)) = get_field(v, "target_resets") { + for item in items.iter().take(64) { + if let toml::Value::Array(fields) = item { + if fields.len() == 2 { + if let (Some(channel), Some(target)) = (toml_u8(&fields[0]), toml_u8(&fields[1])) { + if channel < 2 && (target < 16 || target == 0xff) { + st.pending_target_resets.push_back((channel, (target < 16).then_some(target))); + } + } + } + } + } + } + let u16f = |k: &str| get_field(v, k).and_then(toml_u16).unwrap_or(0); + st.reset_delay1 = u16f("reset_delay1"); + if let Some(m) = get_field(v, "sxp1_bank") { load_u16_slice(m, &mut st.sxp1_bank); } + if let Some(m) = get_field(v, "dma_bank") { load_u16_slice(m, &mut st.dma_bank); } + if let Some(toml::Value::Array(items)) = get_field(v, "outstanding") { + for item in items.iter().take(MAX_REQUESTS as usize) { + if let toml::Value::Array(fields) = item { + if fields.len() == 3 { + if let (Some(target), Some(lun), Some(handle)) = + (toml_u8(&fields[0]), toml_u8(&fields[1]), toml_u32(&fields[2])) { + if target < 32 && lun < 8 { st.outstanding.push((target, lun, handle)); } + } + } + } + } + } + } +} + +/// The two-channel controller. The shared ISP engine owns both buses. +pub struct Isp12160(Isp1040); + +impl Default for Isp12160 { + fn default() -> Self { Self::new(Self::new_bus(), Self::new_bus()) } +} + +impl Isp12160 { + /// Light the status bar's SCSI activity bits in `heartbeat`. + pub fn set_heartbeat(&self, heartbeat: std::sync::Arc) { + self.0.set_heartbeat(heartbeat); + } + + /// A 16-id bus with the ISP controller's disk and CD-ROM identities. + pub fn new_bus() -> ScsiBus { + ScsiBus::new(16, ScsiIdentities { disk: ISP_DISK, cdrom: ISP_CDROM }) + } + + /// The controller with its two channel buses already populated. + pub fn new(bus0: ScsiBus, bus1: ScsiBus) -> Self { + Self(Isp1040::new_12160(bus0, bus1)) + } + + /// Borrow one channel for the shared monitor commands. + pub fn channel(&self, channel: u8) -> Option> { + (channel < 2).then_some(Channel { adapter: self, channel }) + } + + pub fn register_locks(&self) { self.0.register_locks(); } +} + +impl Deref for Isp12160 { + type Target = Core; + fn deref(&self) -> &Core { &self.0 } +} + +/// One channel, with SCSI ids 0–15 as the monitor presents them. +pub struct Channel<'a> { + adapter: &'a Isp12160, + channel: u8, +} + +impl crate::scsi_monitor::ScsiTargets for Channel<'_> { + fn scsi_bus(&self) -> &ScsiBus { + self.adapter.bus_for_channel(self.channel).expect("valid ISP12160 channel") + } +} + +/// The machine API flattens the ISP12160's channels to ids 0–31. +/// Monitor commands use a `Channel` instead, so their ids remain 0–15. +impl DiscChanger for Isp12160 { + fn disc_status(&self) -> Vec<(usize, String, Vec, u64, u64)> { + (0..2).flat_map(|channel| { + crate::scsi_monitor::disc_status(&self.channel(channel).expect("valid ISP12160 channel")) + .into_iter() + .map(move |(id, current, discs, physical, logical)| (id + channel as usize * 16, current, discs, physical, logical)) + }).collect() + } + + fn load_disc(&self, id: usize, path: String) -> Result { + let (channel, target) = Self::split_id(id)?; + crate::scsi_monitor::load_disc(&self.channel(channel).unwrap(), target, path) + } + + fn eject_disc(&self, id: usize) -> Result { + let (channel, target) = Self::split_id(id)?; + crate::scsi_monitor::eject_disc(&self.channel(channel).unwrap(), target) + } + + fn eject_to_empty(&self, id: usize) -> Result<(), String> { + let (channel, target) = Self::split_id(id)?; + crate::scsi_monitor::eject_to_empty(&self.channel(channel).unwrap(), target) + } +} + +impl Isp12160 { + fn split_id(id: usize) -> Result<(u8, usize), String> { + if id >= 32 { return Err(format!("SCSI ID {id} is out of range")); } + Ok(((id / 16) as u8, id % 16)) + } +} + +impl Device for Isp12160 { + fn step(&self, cycles: u64) { Device::step(&self.0, cycles) } + fn start(&self) { Device::start(&self.0) } + fn stop(&self) { Device::stop(&self.0) } + fn is_running(&self) -> bool { Device::is_running(&self.0) } + fn get_clock(&self) -> u64 { Device::get_clock(&self.0) } + + fn register_commands(&self) -> Vec<(String, String)> { + let usage = crate::scsi_monitor::SCSI_USAGE; + vec![ + ("scsi".into(), format!("SCSI (QLogic ISP12160, channel 0): {usage} | scsi notes")), + ("scsi0".into(), "Alias for `scsi` (ISP12160 channel 0)".into()), + ("cow".into(), "COW overlays on ISP12160 channel 0: cow status | cow commit [id] | cow reset [id]".into()), + ("scsi1".into(), format!("SCSI (QLogic ISP12160, channel 1): {usage}")), + ("cow1".into(), "COW overlays on ISP12160 channel 1: same subcommands as `cow`".into()), + ] + } + + fn execute_command(&self, cmd: &str, args: &[&str], mut w: Box) -> Result<(), String> { + use std::io::Write as _; + let (channel, cow) = match cmd { + "scsi" | "scsi0" | "cow" => (0, cmd.starts_with("cow")), + "scsi1" | "cow1" => (1, cmd.starts_with("cow")), + _ => return Err("Command not found".into()), + }; + let ctl = self.channel(channel).expect("valid ISP12160 channel"); + if cow { + return crate::scsi_monitor::cow_command(&ctl, args, &mut *w); + } + if let Some(done) = crate::scsi_monitor::scsi_subcommand(&ctl, args, &mut *w) { + return done; + } + if channel == 0 && args.first().copied() == Some("notes") { + let _ = writeln!(w, "QLogic ISP12160: {} commands", self.total_commands()); + for note in self.notes() { let _ = writeln!(w, " note: {note}"); } + return Ok(()); + } + Err(format!("Usage: {}", crate::scsi_monitor::SCSI_USAGE)) + } +} + +impl Resettable for Isp12160 { + fn power_on(&self) { + self.0.reset(); + } +} + +impl Saveable for Isp12160 { + fn save_state(&self) -> toml::Value { Saveable::save_state(&self.0) } + fn load_state(&self, v: &toml::Value) -> Result<(), String> { Saveable::load_state(&self.0, v) } +} + +impl PciFunction for Isp12160 { + fn state(&self) -> &PciFunctionState { PciFunction::state(&self.0) } + fn bar_read(&self, bar: usize, off: u32, width: usize) -> u32 { PciFunction::bar_read(&self.0, bar, off, width) } + fn bar_write(&self, bar: usize, off: u32, val: u32, width: usize) { PciFunction::bar_write(&self.0, bar, off, val, width) } + fn attached(&self) { PciFunction::attached(&self.0) } + fn reset(&self) { PciFunction::reset(&self.0) } +} + +#[cfg(test)] +mod tests { + use super::*; + + /// The machine disc API uses target 17 for channel 1, target 1. + #[test] + fn disc_status_flattens_the_second_channels_target_ids() { + let bus0 = Isp12160::new_bus(); + let bus1 = Isp12160::new_bus(); + bus1.attach_device(1, "", true, Vec::new(), false, None, false).unwrap(); + let isp = Isp12160::new(bus0, bus1); + + let status = DiscChanger::disc_status(&isp); + assert_eq!(status.iter().map(|drive| drive.0).collect::>(), vec![17]); + } +} diff --git a/src/dev/l1.rs b/src/dev/l1.rs new file mode 100644 index 000000000..c09e9f6a6 --- /dev/null +++ b/src/dev/l1.rs @@ -0,0 +1,454 @@ +//! Fuel's L1 system controller and its processor-side UART. +//! +//! The internal L1 connection is separate from IOC3's external serial ports. +//! The Ns16550 owns the UART registers, FIFO, timeout and interrupt state; +//! this device keeps the controller protocol and its eight-byte register stride. +//! Controller identities and power transitions are functional approximations. +//! Remote routing and controller diagnostics are absent. +//! +//! Ported from hw/misc/sgi-l1.c in jamesbraid/qemu-sgi-origami. + +use crate::dev::ns16550::{com_reg, Ns16550}; +use crate::dev::serial_host::{null_serial_backend, SerialBackend}; +use crate::devlog::LogModule; +use crate::irq::IrqLine; +use crate::traits::{BusDevice, BusRead64, Device, Resettable, BUS_ERR, BUS_OK}; +use parking_lot::Mutex; +use std::collections::VecDeque; +use std::io; +use std::sync::{ + atomic::{AtomicBool, AtomicU64, Ordering}, + Arc, +}; + +mod model; +use model::State; +pub use model::{parse_revision, L1Config, L1Event, DEFAULT_BRICK_SERIAL, DEFAULT_REVISION}; + +fn fcs(mut crc: u16, byte: u8) -> u16 { + crc ^= byte as u16; + for _ in 0..8 { + crc = (crc >> 1) ^ if crc & 1 != 0 { 0x8408 } else { 0 }; + } + crc +} + +fn encode(payload: &[u8]) -> Vec { + let crc = !payload.iter().fold(0xffff, |value, byte| fcs(value, *byte)); + let mut out = vec![0x7e]; + for byte in payload.iter().copied().chain([crc as u8, (crc >> 8) as u8]) { + if matches!(byte, 0x7e | 0x7d | 0x0d) { + out.extend_from_slice(&[0x7d, byte ^ 0x20]); + } else { + out.push(byte); + } + } + out.push(0x7e); + out +} + +const UART_BAUDBASE: u32 = 115200; +const UART_WINDOW_SIZE: u64 = 0x40; + +mod uart_offset { + use super::com_reg; + pub const DATA: u64 = 0x00; + pub const IER: u64 = 0x08; + pub const IIR_FCR: u64 = 0x10; + pub const LCR: u64 = 0x18; + pub const MCR: u64 = 0x20; + pub const LSR: u64 = 0x28; + pub const MSR: u64 = 0x30; + pub const SCR: u64 = 0x38; + + pub fn register(offset: u64) -> u32 { + match offset { + DATA => com_reg::THR, + IER => com_reg::IER, + IIR_FCR => com_reg::IIR_FCR, + LCR => com_reg::LCR, + MCR => com_reg::MCR, + LSR => com_reg::LSR, + MSR => com_reg::MSR, + SCR => com_reg::SCR, + _ => unreachable!("validated L1 UART offset"), + } + } +} + +type EventHandler = Arc; + +#[derive(Default)] +struct Dispatch { + active: bool, + pending: bool, + level: Option, +} + +/// The UART backend does not connect guest L1 commands to a host terminal. +/// Guest transmit bytes are handed to the protocol parser after the UART lock +/// is released; receive bytes come from the protocol reply queue. +struct ProtocolBackend { + guest_tx: Arc>>, + protocol_rx: Arc>>, +} + +impl SerialBackend for ProtocolBackend { + fn send_byte(&self, byte: u8) { + self.guest_tx.lock().push_back(byte); + } + + fn recv_byte(&self) -> io::Result { + self.protocol_rx + .lock() + .pop_front() + .ok_or_else(|| io::ErrorKind::WouldBlock.into()) + } +} + +/// The internal L1 connection is independent of IOC3's external serial ports. +/// Register offsets are relative to the 0x40-byte UART window. Only aligned +/// 64-bit accesses are decoded; register values occupy the low eight bits. +pub struct L1 { + // UART RX takes UART -> protocol queue. Model replies take state -> protocol + // queue. No path holds the protocol queue while acquiring either lock. + state: Arc>, + // Take dispatch before state when draining, release it before the UART or callbacks. + dispatch: Arc>, + uart: Arc>, + guest_tx: Arc>>, + backend: Arc>>, + irq: Arc>, + handler: Arc>>, + running: Arc, + clock: Arc, + worker: Mutex>>, + wake: Arc<(Mutex<()>, parking_lot::Condvar)>, +} +impl L1 { + pub fn new(config: L1Config) -> Result { + if config.rack > 0x3ff || config.bay > 0x3f || config.node_position > 3 { + return Err("L1 rack, bay or node position out of range".into()); + } + if config.mac == [0; 6] || config.mac[0] & 1 != 0 { + return Err("L1 MAC must be nonzero unicast".into()); + } + if config + .brick_serial + .as_ref() + .is_some_and(|s| s.len() > 37 || s.as_bytes().contains(&0)) + { + return Err("L1 brick serial must fit a 37-byte text field".into()); + } + + let protocol_rx = Arc::new(Mutex::new(VecDeque::new())); + let state = Arc::new(Mutex::new(State::new(config, protocol_rx.clone()))); + let guest_tx = Arc::new(Mutex::new(VecDeque::new())); + let protocol_backend = Arc::new(ProtocolBackend { + guest_tx: guest_tx.clone(), + protocol_rx, + }); + let mut uart = Ns16550::new(UART_BAUDBASE); + uart.set_backend(protocol_backend); + + Ok(Self { + state, + dispatch: Arc::new(Mutex::new(Dispatch::default())), + uart: Arc::new(Mutex::new(uart)), + guest_tx, + backend: Arc::new(Mutex::new(null_serial_backend())), + irq: Arc::new(Mutex::new(IrqLine::unconnected())), + handler: Arc::new(Mutex::new(None)), + running: Arc::new(AtomicBool::new(false)), + clock: Arc::new(AtomicU64::new(0)), + worker: Mutex::new(None), + wake: Arc::new((Mutex::new(()), parking_lot::Condvar::new())), + }) + } + + /// Install the separate host-side L1 console connection. + pub fn set_serial_backend(&self, backend: Arc) { + *self.backend.lock() = backend; + } + + pub fn set_irq(&self, line: IrqLine) { + *self.irq.lock() = line; + self.dispatch.lock().level = None; + self.flush(); + } + + /// Power/reset requests reach the machine only after the reply is queued + /// and after the device state lock is released. + pub fn set_event_handler(&self, handler: EventHandler) { + *self.handler.lock() = Some(handler); + } + + #[cfg(test)] + pub fn node_powered(&self) -> bool { + self.state.lock().powered + } + + pub fn station_mac(&self) -> [u8; 6] { + self.state.lock().config.mac + } + + #[cfg(test)] + pub fn config(&self) -> L1Config { + self.state.lock().config.clone() + } + + #[cfg(test)] + pub fn display_line(&self) -> String { + self.state.lock().display.clone() + } + + #[cfg(test)] + pub fn record(&self, component: usize, area: u32) -> Option> { + let index = match area { + 1 => 0, + 2 => 1, + 4 => 2, + _ => return None, + }; + self.state.lock().records.get(component).and_then(|r| { + if r[index].is_empty() { + None + } else { + Some(r[index].clone()) + } + }) + } + + /// Install a raw area before execution. It replaces only the named area; + /// station identity remains the configured MAC, as in the Fuel model. + #[cfg(test)] + pub fn set_record(&self, component: usize, area: u32, bytes: Vec) -> Result<(), String> { + let index = match area { + 1 => 0, + 2 => 1, + 4 => 2, + _ => return Err("unsupported L1 area".into()), + }; + if component >= 16 || bytes.len() > 65535 { + return Err("L1 record out of range".into()); + } + self.state.lock().records[component][index] = bytes; + Ok(()) + } + + pub fn send_event(&self, code: u32, number: u32, text: &str) { + self.state.lock().event(code, number, text); + self.service_protocol_uart(); + self.flush(); + } + + /// Deterministic input path shared with the polled host serial backend. + pub fn receive_console(&self, bytes: &[u8]) { + for &byte in bytes { + self.state.lock().console_input(byte); + self.service_protocol_uart(); + self.flush(); + } + } + + pub fn poll_serial(&self) { + let backend = self.backend.lock().clone(); + for _ in 0..256 { + if self.state.lock().queue.lock().len() > 4088 { + break; + } + match backend.recv_byte() { + Ok(byte) => self.receive_console(&[byte]), + Err(error) => { + if error.kind() != io::ErrorKind::WouldBlock { + log::debug!("L1 serial input: {error}"); + } + break; + } + } + } + } + + /// Let the native UART pull reply bytes from the model's shared bounded queue. + fn service_protocol_uart(&self) { + let mut uart = self.uart.lock(); + uart.pump(); + uart.check_timeout(std::time::Instant::now()); + } + + /// Parse guest transmit only after the UART lock is released. The UART + /// callback only appends to this independent queue. + fn process_guest_transmit(&self) { + let bytes: Vec<_> = self.guest_tx.lock().drain(..).collect(); + if bytes.is_empty() { + return; + } + { + let mut state = self.state.lock(); + for byte in bytes { + state.parse(byte); + } + } + self.service_protocol_uart(); + } + + fn flush(&self) { + let mut dispatch = self.dispatch.lock(); + dispatch.pending = true; + if dispatch.active { + return; + } + dispatch.active = true; + loop { + dispatch.pending = false; + let (output, actions) = { + let mut state = self.state.lock(); + ( + std::mem::take(&mut state.output), + std::mem::take(&mut state.actions), + ) + }; + drop(dispatch); + let level = self.uart.lock().irq(); + + // Reentrant reads request another drain, but must not repeat an + // unchanged IRQ and recursively trigger the same machine callback. + dispatch = self.dispatch.lock(); + let changed = dispatch.level != Some(level); + dispatch.level = Some(level); + drop(dispatch); + + if changed { + let irq = self.irq.lock().clone(); + irq.set(level); + } + let backend = self.backend.lock().clone(); + for byte in output { + backend.send_byte(byte); + } + let handler = self.handler.lock().clone(); + for action in actions { + if let Some(handler) = &handler { + handler(action); + } else { + crate::dlog!(LogModule::L1, "unconnected L1 action {action:?}"); + } + } + + dispatch = self.dispatch.lock(); + if !dispatch.pending { + dispatch.active = false; + break; + } + } + } +} +impl BusDevice for L1 { + fn read64(&self, addr: u64) -> BusRead64 { + if addr >= UART_WINDOW_SIZE || addr & 7 != 0 { + return BusRead64::err(); + } + let value = self.uart.lock().read(uart_offset::register(addr)); + self.flush(); + BusRead64::ok(value as u64) + } + + fn write64(&self, addr: u64, value: u64) -> u32 { + if addr >= UART_WINDOW_SIZE || addr & 7 != 0 { + return BUS_ERR; + } + let register = uart_offset::register(addr); + let value = value as u8; + if register == com_reg::IIR_FCR && value & com_reg::FCR_RFR != 0 { + self.state.lock().queue.lock().clear(); + } + self.uart.lock().write(register, value); + self.process_guest_transmit(); + self.flush(); + BUS_OK + } +} +impl Resettable for L1 { + fn power_on(&self) { + self.state.lock().reset(); + self.guest_tx.lock().clear(); + self.uart.lock().reset(); + self.clock.store(0, Ordering::Relaxed); + self.flush(); + } +} +impl Device for L1 { + fn start(&self) { + let mut worker = self.worker.lock(); + if worker.is_some() { + return; + } + self.running.store(true, Ordering::Release); + let device = Self { + state: self.state.clone(), + dispatch: self.dispatch.clone(), + uart: self.uart.clone(), + guest_tx: self.guest_tx.clone(), + backend: self.backend.clone(), + irq: self.irq.clone(), + handler: self.handler.clone(), + running: self.running.clone(), + clock: self.clock.clone(), + worker: Mutex::new(None), + wake: self.wake.clone(), + }; + *worker = Some( + std::thread::Builder::new() + .name("l1-serial".into()) + .spawn(move || { + while device.is_running() { + device.poll_serial(); + let mut guard = device.wake.0.lock(); + if device.is_running() { + device + .wake + .1 + .wait_for(&mut guard, std::time::Duration::from_millis(2)); + } + } + }) + .expect("start L1 serial worker"), + ); + } + + fn stop(&self) { + let mut slot = self.worker.lock(); + { + let _guard = self.wake.0.lock(); + self.running.store(false, Ordering::Release); + self.wake.1.notify_all(); + } + let worker = slot.take(); + if let Some(worker) = worker { + if worker.thread().id() != std::thread::current().id() { + let _ = worker.join(); + } + } + } + + fn is_running(&self) -> bool { + self.running.load(Ordering::Acquire) + } + + fn get_clock(&self) -> u64 { + self.clock.load(Ordering::Relaxed) + } + + fn step(&self, cycles: u64) { + if self.is_running() { + self.clock.fetch_add(cycles, Ordering::Relaxed); + } + } +} +impl Drop for L1 { + fn drop(&mut self) { + self.stop(); + } +} +#[cfg(test)] +mod tests; diff --git a/src/dev/l1/model.rs b/src/dev/l1/model.rs new file mode 100644 index 000000000..907fd59dd --- /dev/null +++ b/src/dev/l1/model.rs @@ -0,0 +1,764 @@ +//! Fuel's L1 controller protocol parser and reply model. +//! +//! This state owns packet framing, controller identity records, console +//! commands and power events. The UART register file and receive pacing live +//! in the shared Ns16550 core. +//! +//! Ported from hw/misc/sgi-l1.c in jamesbraid/qemu-sgi-origami. + +use super::*; + +/// Firmware revision the model reports for request 0x0005. +/// +/// Model choice: 1.48.1 is the revision a retained real Fuel L1 session +/// prints (the Fuel contract's L1 section). The model reproduces that +/// firmware's replies, not its code, so the revision is a label. +pub const DEFAULT_REVISION: [u32; 3] = [1, 48, 1]; + +/// Brick serial the model reports for request 0x0004. +/// +/// Model choice: the marked synthetic text form, system-board letter `N`, +/// instance 0 (the component taxonomy's synthetic serials). The real format +/// is unknown, and the value must not pass as a real SGI serial. +pub const DEFAULT_BRICK_SERIAL: &str = "EMUN00"; + +/// Parse a "major.minor.patch" revision string, as a `[fuel] l1_revision` +/// override spells it. Each part is a decimal number. +pub fn parse_revision(text: &str) -> Result<[u32; 3], String> { + let bad = || format!("L1 revision \"{text}\" must be major.minor.patch, each a decimal number"); + let mut parts = text.split('.'); + let mut rev = [0u32; 3]; + for slot in &mut rev { + let part = parts.next().ok_or_else(bad)?; + if part.is_empty() || !part.bytes().all(|b| b.is_ascii_digit()) { + return Err(bad()); + } + *slot = part.parse().map_err(|_| bad())?; + } + if parts.next().is_some() { + return Err(bad()); + } + Ok(rev) +} + +#[derive(Clone, Debug)] +pub struct L1Config { + pub rack: u32, + pub bay: u32, + pub node_position: u8, + pub type_code: u32, + pub revision: Option<[u32; 3]>, + pub mac: [u8; 6], + pub brick_serial: Option, + pub auto_power: bool, +} +impl Default for L1Config { + fn default() -> Self { + Self { + rack: 1, + bay: 1, + node_position: 0, + type_code: 0x41, + revision: Some(DEFAULT_REVISION), + mac: [8, 0, 0x69, 0x12, 0x34, 0x56], + brick_serial: Some(DEFAULT_BRICK_SERIAL.into()), + auto_power: true, + } + } +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum L1Event { + Power(bool), + Reset, + SoftReset, + Nmi, + Shutdown, +} + +pub(super) struct State { + pub config: L1Config, + pub queue: Arc>>, + frame: Vec, + in_frame: bool, + escape: bool, + overflow: bool, + pub display: String, + pub records: [[Vec; 3]; 16], + pub actions: Vec, + pub output: Vec, + pub powered: bool, + console: bool, + escaped: bool, + command: Vec, + port: u8, + debug: u32, + partition: u32, + two_bit: u32, + reasons: [u32; 2], + events: Option, + power_down_pending: bool, +} +fn text_field(out: &mut Vec, text: &[u8]) { + out.push(1); + out.extend_from_slice(text); + out.push(0); +} +pub(super) fn ints(values: &[u32]) -> Vec { + let mut out = Vec::new(); + for v in values { + out.push(0); + out.extend_from_slice(&v.to_be_bytes()); + } + out +} +fn word(p: &[u8]) -> u32 { + u32::from_be_bytes(p[..4].try_into().unwrap()) +} +fn arguments(mut p: &[u8], count: u8) -> Option> { + let mut fields = Vec::new(); + for _ in 0..count { + let tag = *p.first()?; + let len = match tag { + 0 => 5, + 1 => p[1..].iter().position(|b| *b == 0)? + 2, + t if t & 0x80 != 0 => 1 + (t & 0x7f) as usize, + _ => return None, + }; + fields.push(p.get(..len)?); + p = &p[len..]; + } + if p.is_empty() { + Some(fields) + } else { + None + } +} +fn area_finish(mut area: Vec) -> Vec { + while (area.len() + 1) % 8 != 0 { + area.push(0); + } + let sum = area.iter().fold(0u8, |s, b| s.wrapping_add(*b)); + area.push(sum.wrapping_neg()); + area +} +fn ia_text(area: &mut Vec, text: &str) { + area.push(0xc0 | text.len() as u8); + area.extend_from_slice(text.as_bytes()); +} +fn board(mac: Option<[u8; 6]>) -> Vec { + let mut b = vec![0; 6]; + for s in ["SYNTHETIC", "IP34", "EMUN00", "030-1707-003"] { + ia_text(&mut b, s); + } + b.push(0); + ia_text(&mut b, "-G"); + b.extend_from_slice(&[1, 0]); + ia_text(&mut b, "00"); + for _ in 0..3 { + b.extend_from_slice(&[4, 0, 0, 0, 0]); + } + if let Some(mac) = mac { + ia_text( + &mut b, + &mac.iter().map(|v| format!("{v:02X}")).collect::(), + ); + } + b.push(0xc1); + // The count includes the checksum and padding, not a separate header. + b[1] = (b.len() + 1).div_ceil(8) as u8; + area_finish(b) +} +impl State { + pub fn new(config: L1Config, queue: Arc>>) -> Self { + let mut records: [[Vec; 3]; 16] = + std::array::from_fn(|_| std::array::from_fn(|_| Vec::new())); + let chassis = area_finish(vec![0, 2, 0, 0xc2, b'N', b'A', 0xc2, b'N', b'A', 0xc1]); + for c in 0..2 { + records[c][0] = chassis.clone(); + records[c][1] = board(if c == 1 { Some(config.mac) } else { None }); + } + let powered = config.auto_power; + let port = config.node_position << 5; + Self { + config, + queue, + frame: Vec::new(), + in_frame: false, + escape: false, + overflow: false, + display: "L1 running".into(), + records, + actions: Vec::new(), + output: Vec::new(), + powered, + console: powered, + escaped: false, + command: Vec::new(), + port, + debug: 0, + partition: 0, + two_bit: 0, + reasons: [0; 2], + events: None, + power_down_pending: false, + } + } + pub fn reset(&mut self) { + // Controller settings survive the controller's local boot. The UART and + // packet parser reset independently of those persistent settings. + let persistent = (self.debug, self.partition, self.two_bit, self.reasons); + let events = self.events.filter(|&channel| channel >= 4); + let records = std::mem::replace( + &mut self.records, + std::array::from_fn(|_| std::array::from_fn(|_| Vec::new())), + ); + let queue = self.queue.clone(); + queue.lock().clear(); + *self = Self::new(self.config.clone(), queue); + (self.debug, self.partition, self.two_bit, self.reasons) = persistent; + self.events = events; + self.records = records; + } + fn emit(&mut self, payload: &[u8]) { + if payload.len() > 128 { + crate::dlog!(LogModule::L1, "L1 reply exceeds frame capacity"); + return; + } + let bytes = encode(payload); + let mut queue = self.queue.lock(); + if queue.len() + bytes.len() <= 4096 { + queue.extend(bytes); + } else { + crate::dlog!(LogModule::L1, "L1 receive queue full"); + } + } + fn reply(&mut self, channel: u8, code: i32, count: u8, fields: &[u8]) { + let mut p = vec![0x20 | channel]; + p.extend_from_slice(&code.to_be_bytes()); + p.push(count); + p.extend_from_slice(fields); + self.emit(&p); + } + pub fn parse(&mut self, mut byte: u8) { + if byte == 0x7e { + if self.in_frame + && !self.escape + && !self.overflow + && self.frame.len() >= 3 + && self.frame.iter().fold(0xffff, |v, b| fcs(v, *b)) == 0xf0b8 + { + let payload = self.frame[..self.frame.len() - 2].to_vec(); + self.request(&payload); + } + self.in_frame = true; + self.escape = false; + self.overflow = false; + self.frame.clear(); + return; + } + if !self.in_frame || self.overflow { + return; + } + if byte == 0x7d { + self.escape = true; + return; + } + if self.escape { + byte ^= 0x20; + self.escape = false; + } + if self.frame.len() == 258 { + self.overflow = true; + } else { + self.frame.push(byte); + } + } + fn eeprom(&mut self, channel: u8, a: &[&[u8]]) { + if a.len() < 4 || a[..4].iter().any(|v| v[0] != 0) { + self.reply(channel, -100, 0, &[]); + return; + } + let component = word(&a[0][1..]) as usize; + let area = word(&a[1][1..]); + let offset = word(&a[2][1..]) as usize; + let requested = word(&a[3][1..]) as usize; + if component >= 16 || self.records[component].iter().all(Vec::is_empty) { + self.reply(channel, -104, 0, &[]); + return; + } + if area > 4 { + self.reply(channel, -105, 0, &[]); + return; + } + let index = match area { + 1 => 0, + 2 => 1, + 4 => 2, + _ => { + self.reply(channel, -104, 0, &[]); + return; + } + }; + let record = &self.records[component][index]; + if record.is_empty() { + self.reply(channel, -104, 0, &[]); + return; + } + let checksum_len = if area == 4 { + record.len().min(64) + } else { + record.len() + }; + if record[..checksum_len] + .iter() + .fold(0u8, |s, b| s.wrapping_add(*b)) + != 0 + { + self.reply(channel, -107, 0, &[]); + return; + } + if offset >= record.len() { + self.reply(channel, -105, 0, &[]); + return; + } + let len = requested.min(record.len() - offset); + if len > 116 { + crate::dlog!(LogModule::L1, "L1 EEPROM reply exceeds frame capacity"); + self.reply(channel, -105, 0, &[]); + return; + } + let mut fields = ints(&[(record.len() - offset - len) as u32]); + if requested != 0 { + fields.push(0x80 | len as u8); + fields.extend_from_slice(&record[offset..offset + len]); + } + self.reply(channel, 0, if requested == 0 { 1 } else { 2 }, &fields); + } + fn request(&mut self, p: &[u8]) { + let channel = p[0] & 31; + if p[0] & 0xe0 == 0x40 { + self.port = (self.port & 0x60) | channel; + if self.console && self.powered { + self.output.extend_from_slice(&p[1..]); + } else { + crate::dlog!(LogModule::L1, "L1 console output while operator is in L1 mode"); + } + return; + } + if p[0] & 0xe0 != 0 || p.len() < 7 { + return; + } + let address = word(&p[1..]); + let cmd = u16::from_be_bytes([p[5], p[6]]); + let key = address >> 12; + if key != 0xffff && key != ((self.config.rack << 6) | self.config.bay) { + crate::dlog!(LogModule::L1, "L1 unsupported destination {address:#x}"); + self.emit(&[0x20 | channel, 255, 255, 255, 255, 0, 0, 0, 8]); + return; + } + let count = if p.len() == 7 { 0 } else { p[7] }; + let raw = if p.len() > 8 { &p[8..] } else { &[] }; + let task = address & 31; + if task == 5 || task == 21 { + if self.powered { + self.emit(&[ + ((address >> 5) & 31) as u8, + raw.get(4).copied().unwrap_or(0), + ]); + } else { + let mut field = Vec::new(); + text_field( + &mut field, + format!( + "WARNING: power on {:03}c{:02} appears off!", + self.config.rack, self.config.bay + ) + .as_bytes(), + ); + self.reply(channel, -1, 1, &field); + } + return; + } + if task != 6 && task != 3 && task != 18 { + crate::dlog!(LogModule::L1, "L1 unsupported task {task}"); + return; + } + let Some(a) = arguments(raw, count) else { + self.reply(channel, -100, 0, &[]); + return; + }; + if task == 3 { + if a.len() != 1 || a[0][0] != 1 { + self.reply(channel, -100, 0, &[]); + return; + } + let text = String::from_utf8_lossy(&a[0][1..a[0].len() - 1]); + let (code, output) = self.run_line(&text, false); + let mut fields = Vec::new(); + let mut n = 0; + for line in output.split("\r\n").filter(|s| !s.is_empty()) { + if fields.len() + line.len() + 2 > 121 { + break; + } + text_field(&mut fields, line.as_bytes()); + n += 1; + } + self.reply(channel, code, n, &fields); + return; + } + if task == 18 { + let length = a.iter().take(3).map(|v| v.len()).sum::().min(121); + self.reply(channel, cmd as i32, count.min(3), &raw[..length]); + return; + } + let mut values = Vec::new(); + let mut fields = Vec::new(); + let mut field_count = 0; + let mut error = 0; + match cmd { + 1 => values.push(self.debug), + 2 => values.push((self.config.rack << 6) | self.config.bay), + 3 => { + if self.config.type_code == 0 { + error = -104; + } else { + values.extend([ + (self.config.rack << 6) | self.config.bay, + self.config.type_code, + ]); + } + } + 4 => { + if let Some(serial) = &self.config.brick_serial { + text_field(&mut fields, serial.as_bytes()); + field_count = 1; + } else { + error = -104; + } + } + 5 => { + if let Some(rev) = self.config.revision { + values.extend(rev); + } else { + crate::dlog!(LogModule::L1, "L1 revision metadata not configured"); + error = -104; + } + } + 6 => { + self.eeprom(channel, &a); + return; + } + 7 => error = if a.is_empty() { -100 } else { -104 }, + // The system serial is the station address as twelve uppercase + // hexadecimal digits in address order, no separators or OUI + // added: the text the stock PROM converts straight back into the + // Ethernet address (the Fuel contract's MAC section). The + // uppercase case is the contract's choice for the model. + 8 => { + let mac = self + .config + .mac + .iter() + .map(|b| format!("{b:02X}")) + .collect::(); + text_field(&mut fields, mac.as_bytes()); + field_count = 1; + } + 9 => values.push(self.partition), + 0xa => values.extend([0, 0]), + 0xb => values.push(self.two_bit), + 0xc => values.extend([0; 8]), + 0xd => values.extend([0; 4]), + 0xe => values.push((self.port >> 5) as u32 & 3), + 0xf => values.extend(self.reasons), + 0x10 => values.extend([1, 1, 1, 1]), + 0x11 => { + error = -106; + values.push(1); + } + 0x1001 | 0x1002 | 0x1006 | 0x1007 | 0x1008 => { + if a.is_empty() { + error = -100; + } else if a[0][0] != 0 { + error = -105; + } else { + let v = word(&a[0][1..]); + match cmd { + 0x1001 => self.debug = v & 0xffff, + 0x1002 => self.port = ((self.port as u32 & 0x60) | v) as u8 & 0x7f, + 0x1006 => self.partition = v, + 0x1007 => self.events = Some(v as u8), + _ => self.two_bit = v & 3, + } + } + } + 0x1003 => {} + 0x1004 | 0x1005 => { + if a.is_empty() { + error = -100; + } else if a[0][0] != 1 { + error = -102; + } else { + self.display = a[0][1..a[0].len() - 1] + .iter() + .take(72) + .map(|b| { + if (0x20..=0x7e).contains(b) { + *b as char + } else { + '?' + } + }) + .collect(); + } + } + 0x1009 => { + if a.len() < 2 { + error = -100; + } else if a[0][0] != 0 || a[1][0] != 0 || word(&a[1][1..]) & 15 > 2 { + error = -105; + } + } + 0x100a => self.events = None, + 0x100b => { + if a.len() != 3 { + error = -100; + } else { + self.events = None; + } + } + 0x100c => { + if a.len() != 2 { + error = -100; + } + } + 0x2001 | 0x2005 | 0x2006 => { + self.reply(channel, 0, 0, &[]); + match cmd { + 0x2001 => self.reset_node(), + 0x2005 => self.power_down(true, false), + _ => { + self.power_down(false, false); + self.power_up(true); + } + } + return; + } + 0x2002..=0x2004 => error = -100, + 0x2007 | 0x2008 => error = -108, + 0x4000 => error = -104, + 0x4002 => { + self.reply(channel, 0, 0, &[]); + self.reboot(); + return; + } + _ => { + crate::dlog!(LogModule::L1, "L1 unsupported request {cmd:#x}"); + error = -101; + } + } + if !values.is_empty() { + field_count = values.len() as u8; + fields = ints(&values); + } + self.reply(channel, error, field_count, &fields); + } + fn reason(&mut self) { + self.reasons[1] = self.reasons[0]; + self.reasons[0] = 3; + } + fn reset_node(&mut self) { + if self.powered { + self.reason(); + self.display = "Reset done".into(); + self.actions.push(L1Event::Reset); + } + } + fn power_down(&mut self, shutdown: bool, confirm: bool) { + if self.powered { + if confirm && !self.power_down_pending { + self.power_down_pending = true; + self.event( + 0x8000, + 0, + "Power Down: issue 'pwr d' again to immediately power down.", + ); + return; + } + self.power_down_pending = false; + self.reason(); + self.powered = false; + self.display = "Powered Down".into(); + self.actions.push(L1Event::Power(false)); + if shutdown { + self.actions.push(L1Event::Shutdown); + } + } + } + fn power_up(&mut self, restart: bool) { + self.reason(); + self.powered = true; + self.power_down_pending = false; + self.port &= 0x60; + self.console = true; + self.display = "Powered Up".into(); + self.actions.push(L1Event::Power(true)); + if restart { + self.actions.push(L1Event::Reset); + } + } + fn reboot(&mut self) { + self.power_down_pending = false; + self.console = false; + self.escaped = false; + self.command.clear(); + self.frame.clear(); + self.in_frame = false; + self.port = self.config.node_position << 5; + self.display = "L1 running".into(); + if self.events.is_some_and(|c| c < 4) { + self.events = None; + } + } + pub fn event(&mut self, code: u32, number: u32, text: &str) { + let Some(channel) = self.events else { + return; + }; + let mut p = vec![0x40 | (channel & 31)]; + p.extend_from_slice( + &((self.config.rack << 18) | (self.config.bay << 12) | 4).to_be_bytes(), + ); + p.extend_from_slice(&code.to_be_bytes()); + p.push(2); + p.extend(ints(&[number])); + text_field( + &mut p, + format!( + "\r\n{:03}c{:02} ATTN: {text}\r\n", + self.config.rack, self.config.bay + ) + .as_bytes(), + ); + if p.len() <= 127 { + self.emit(&p); + } + } + fn run_line(&mut self, line: &str, interactive: bool) -> (i32, String) { + let words = line.split_whitespace().collect::>(); + let local = if words.first() == Some(&"*") { + &words[1..] + } else { + &words[..] + }; + match local { + ["pwr" | "power", "d" | "down"] => { + self.power_down(true, interactive); + (0, String::new()) + } + ["pwr" | "power", "u" | "up"] => { + if !self.powered { + self.power_up(true); + } + (0, String::new()) + } + ["reset"] => { + self.reset_node(); + (0, String::new()) + } + ["softreset"] => { + self.actions.push(L1Event::SoftReset); + self.display = "Sft rst done".into(); + (0, String::new()) + } + ["nmi"] => { + self.actions.push(L1Event::Nmi); + self.display = "NMI done".into(); + (0, String::new()) + } + ["reboot_l1"] => { + self.reboot(); + (0, String::new()) + } + ["l1"] => { + self.escaped = false; + (0, String::new()) + } + ["version"] => { + if let Some(r) = self.config.revision { + ( + 0, + format!("L1 Controller: Revision {}.{}.{}\r\n", r[0], r[1], r[2]), + ) + } else { + ( + -104, + "L1 firmware revision metadata not configured\r\n".into(), + ) + } + } + ["debug"] if words.len() == 1 => ( + 0, + format!("debug switches set to 0x{:04x}\n\r\n", self.debug), + ), + ["debug", v] if words.len() == 2 => { + let n = if let Some(hex) = v.strip_prefix("0x") { + u32::from_str_radix(hex, 16) + } else { + v.parse() + }; + match n { + Ok(v) => { + self.debug = v & 0xffff; + ( + 0, + format!("debug switches set to 0x{:04x}\n\r\n", self.debug), + ) + } + Err(_) => (-1200, "ERROR: command not found.\n\r\n".into()), + } + } + _ => { + crate::dlog!(LogModule::L1, "L1 unsupported operator command {line:?}"); + (-1, "ERROR: command not found.\r\n".into()) + } + } + } + fn prompt(&mut self) { + self.output.extend_from_slice( + format!("{:03}c{:02}-L1>", self.config.rack, self.config.bay).as_bytes(), + ); + } + pub fn console_input(&mut self, byte: u8) { + if self.console { + if byte == 0x14 { + self.console = false; + self.escaped = true; + self.output + .extend_from_slice(b"escaping to L1 system controller\r\n"); + self.prompt(); + } else if self.powered { + self.emit(&[self.port & 31, byte]); + } else { + crate::dlog!(LogModule::L1, "L1 console input while powered off"); + } + } else if byte == 4 { + self.command.clear(); + self.console = true; + } else if byte == b'\r' || byte == b'\n' { + let line = String::from_utf8_lossy(&self.command).into_owned(); + self.command.clear(); + let (_, output) = self.run_line(&line, true); + self.output.extend_from_slice(output.as_bytes()); + if self.escaped { + self.console = true; + } else { + self.prompt(); + } + } else if byte == 8 || byte == 127 { + self.command.pop(); + } else if self.command.len() < 255 { + self.command.push(byte); + } + } +} diff --git a/src/dev/l1/tests.rs b/src/dev/l1/tests.rs new file mode 100644 index 000000000..64d206fd5 --- /dev/null +++ b/src/dev/l1/tests.rs @@ -0,0 +1,381 @@ +// Cases follow tests/qtest/sn1-l1-test.c in jamesbraid/qemu-sgi-origami: +// uart, frames, identity, eeprom, errors and reset. + +use super::*; + +fn send(l1: &L1, bytes: &[u8]) { + for &b in bytes { + assert_eq!(l1.write64(0, b as u64), BUS_OK); + } +} +fn drain(l1: &L1) -> Vec { + let mut out = Vec::new(); + while l1.read64(0x28).data & 1 != 0 { + out.push(l1.read64(0).data as u8); + } + out +} +fn request(l1: &L1, command: u16, args: &[u8], count: u8) -> Vec { + let mut p = vec![ + 0x10, + 0x0f, + 0xff, + 0xf0, + 6, + (command >> 8) as u8, + command as u8, + count, + ]; + p.extend_from_slice(args); + send(l1, &encode(&p)); + drain(l1) +} +fn integer_reply(values: &[u32]) -> Vec { + let mut p = vec![0x30, 0, 0, 0, 0, values.len() as u8]; + p.extend(model::ints(values)); + encode(&p) +} + +/// The PROM accesses the L1 UART through byte-sized register operations. +#[test] +fn the_l1_uart_frames_bytes_and_prioritizes_interrupts() { + let l1 = L1::new(Default::default()).unwrap(); + assert_eq!(l1.read64(1).status, BUS_ERR); + assert_eq!(l1.read8(0).status, BUS_ERR); + assert_eq!(l1.read32(0).status, BUS_ERR); + assert_eq!(l1.read64(0x28).data, 0x60); + send(&l1, &[0x7e, 0x10, 0x0f, 0xff, 0xf0, 6, 0, 0x0e, 0, 0xf9, 0x72, 0x7e]); + assert_eq!(drain(&l1), [0x7e, 0x30, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0x30, 0x98, 0x7e]); + let (irq, line) = crate::irq::LevelProbe::line(); + l1.set_irq(line); + l1.write64(8, 3); + assert!(irq.level()); + assert_eq!(l1.read64(0x10).data, 2); + assert!(!irq.level()); + l1.write64(0x18, 0x80); + l1.write64(0, 42); + l1.write64(8, 7); + assert_eq!(l1.read64(0).data, 42); + assert_eq!(l1.read64(8).data, 7); + l1.write64(0x18, 3); + send(&l1, &encode(&[0x10, 0x0f, 0xff, 0xf0, 6, 0, 0x0e, 0])); + assert_eq!(l1.read64(0x10).data, 4); + drain(&l1); + assert_eq!(l1.read64(0x10).data, 2); + assert!(!irq.level()); +} + +/// The guest reads controller identity and checks the EEPROM record checksum. +#[test] +fn the_l1_returns_identity_and_validated_eeprom_records() { + let l1 = L1::new(Default::default()).unwrap(); + assert_eq!(request(&l1, 3, &[], 0), integer_reply(&[65, 65])); + let board = l1.record(1, 2).unwrap(); + assert_eq!(board.iter().fold(0u8, |s, x| s.wrapping_add(*x)), 0); + for text in [b"IP34".as_slice(), b"EMUN00", b"080069123456"] { + assert!(board.windows(text.len()).any(|x| x == text)); + } + let mut expected = vec![0x30, 0, 0, 0, 0, 2]; + expected.extend(model::ints(&[0])); + expected.push(0x80 | board.len() as u8); + expected.extend(board); + assert_eq!(request(&l1, 6, &model::ints(&[1, 2, 0, 116]), 4), encode(&expected)); + for (args, count, error) in [([2, 5, 0, 1], 4, 152), ([1, 5, 0, 1], 4, 151), ([1, 2, 0, 1], 3, 156)] { + assert_eq!(request(&l1, 6, &model::ints(&args), count), encode(&[0x30, 255, 255, 255, error, 0])); + } + l1.power_on(); + assert_eq!(request(&l1, 5, &[], 0), integer_reply(&[1, 48, 1])); +} + +/// A successful reply carrying one text field. +fn text_reply(text: &str) -> Vec { + let mut p = vec![0x30, 0, 0, 0, 0, 1]; + p.push(1); + p.extend_from_slice(text.as_bytes()); + p.push(0); + encode(&p) +} + +/// With no configuration the controller reports the labelled model defaults: +/// firmware 1.48.1 and the synthetic brick serial. +#[test] +fn the_l1_reports_the_default_revision_and_brick_serial() { + let l1 = L1::new(Default::default()).unwrap(); + l1.power_on(); + assert_eq!(request(&l1, 5, &[], 0), integer_reply(&[1, 48, 1])); + assert_eq!(request(&l1, 4, &[], 0), text_reply("EMUN00")); +} + +/// Both identity values can be replaced. +#[test] +fn the_l1_revision_and_brick_serial_can_be_overridden() { + let l1 = L1::new(L1Config { + revision: Some(parse_revision("2.13.40").unwrap()), + brick_serial: Some("EMUN07".into()), + ..Default::default() + }) + .unwrap(); + l1.power_on(); + assert_eq!(request(&l1, 5, &[], 0), integer_reply(&[2, 13, 40])); + assert_eq!(request(&l1, 4, &[], 0), text_reply("EMUN07")); + assert_eq!( + L1::new(L1Config { brick_serial: Some("x".repeat(38)), ..Default::default() }).err().as_deref(), + Some("L1 brick serial must fit a 37-byte text field") + ); +} + +/// The revision override is exactly three decimal numbers. +#[test] +fn the_l1_revision_text_is_three_decimal_parts() { + assert_eq!(parse_revision("1.48.1"), Ok([1, 48, 1])); + for bad in ["", "1", "1.48", "1.48.1.0", "1..1", "a.b.c", "1.48.-1", "1.48.+1", " 1.48.1", "1.48.99999999999"] { + assert!(parse_revision(bad).is_err(), "{bad:?}"); + } +} + +/// Request 0x0008 is the configured station address as twelve uppercase +/// hexadecimal digits in address order, the text the PROM turns back into the +/// Ethernet address. +#[test] +fn the_l1_system_serial_is_the_station_address_in_hex() { + let l1 = L1::new(Default::default()).unwrap(); + l1.power_on(); + assert_eq!(request(&l1, 8, &[], 0), text_reply("080069123456")); + let l1 = L1::new(L1Config { mac: [0x00, 0xA0, 0xc9, 0x1f, 0x02, 0xbe], ..Default::default() }).unwrap(); + l1.power_on(); + assert_eq!(request(&l1, 8, &[], 0), text_reply("00A0C91F02BE")); +} + +/// The L1 parser recovers after malformed frames and preserves escaped console text. +#[test] +fn the_l1_resynchronizes_after_bad_frames_and_preserves_escaped_text() { + let l1 = L1::new(Default::default()).unwrap(); + send(&l1, &[0x7e, 1, 2, 3, 0x7e]); + send(&l1, &vec![0x41; 259]); + send(&l1, &[0x7e]); + assert!(drain(&l1).is_empty()); + assert_eq!(request(&l1, 0x2000, &[], 0), encode(&[0x30, 255, 255, 255, 155, 0])); + send(&l1, &encode(&[0x10, 0x4f, 0xff, 0xf0, 6, 0, 3, 0])); + assert_eq!(drain(&l1), encode(&[0x30, 255, 255, 255, 255, 0, 0, 0, 8])); + let mut text = vec![1]; + text.extend_from_slice(b"hi~}\r\0"); + assert_eq!(request(&l1, 0x1004, &text, 1), integer_reply(&[])); + assert_eq!(l1.display_line(), "hi~}?"); + text.truncate(1); + text.extend_from_slice(b"abcdefghijkl\x01\x7f"); + text.extend(std::iter::repeat_n(b'z', 70)); + text.push(0); + assert_eq!(request(&l1, 0x1004, &text, 1), integer_reply(&[])); + assert_eq!(l1.display_line(), format!("abcdefghijkl??{}", "z".repeat(58))); +} + +/// The PROM may reset L1 during boot; controller settings and channel configuration must remain available. +#[test] +fn l1_reset_clears_uart_state_but_preserves_settings_and_channels() { + for channel in [3, 23] { + let l1 = L1::new(Default::default()).unwrap(); + let (irq, line) = crate::irq::LevelProbe::line(); + l1.set_irq(line); + for (set, value) in [(0x1001, 0x1234), (0x1006, 42), (0x1008, 2), (0x1007, channel)] { + request(&l1, set, &model::ints(&[value]), 1); + } + request(&l1, 0x2001, &[], 0); + let reasons = request(&l1, 0xf, &[], 0); + assert_eq!(reasons, integer_reply(&[3, 0])); + l1.write64(8, 1); + l1.send_event(0x8000, 7, "before reset"); + assert!(irq.level()); + let stale = encode(&[0x10, 0x0f, 0xff, 0xf0, 6, 0, 9, 0]); + send(&l1, &stale[..4]); + l1.write64(0x18, 0x80); + l1.write64(0, 42); + l1.write64(8, 7); + l1.power_on(); + assert!(!irq.level()); + assert_eq!(l1.read64(0x18).data, 0); + assert_eq!(l1.read64(8).data, 0); + assert!(drain(&l1).is_empty()); + l1.write64(0x18, 0x80); + // Ns16550 resets its divisor latch to 0x000c for 9600 baud. + assert_eq!(l1.read64(0).data, 0x0c); + assert_eq!(l1.read64(8).data, 0); + l1.write64(0x18, 0); + send(&l1, &stale[4..]); + assert!(drain(&l1).is_empty()); + for (get, value) in [(1, 0x1234), (9, 42), (0xb, 2)] { + assert_eq!(request(&l1, get, &[], 0), integer_reply(&[value])); + } + assert_eq!(request(&l1, 0xf, &[], 0), reasons); + l1.write64(8, 1); + l1.send_event(0x8000, 7, "after reset"); + assert_eq!(irq.level(), channel == 23); + let emitted = drain(&l1); + assert_eq!(emitted.is_empty(), channel == 3); + assert!(!irq.level()); + } +} + +/// A local console power-down request requires a second confirmation command. +#[test] +fn interactive_power_down_waits_for_a_second_command() { + let l1 = Arc::new(L1::new(Default::default()).unwrap()); + let actions = Arc::new(Mutex::new(Vec::new())); + let observed = actions.clone(); + let weak = Arc::downgrade(&l1); + l1.set_event_handler(Arc::new(move |event| { + assert!(!weak.upgrade().unwrap().node_powered()); + observed.lock().push(event); + })); + request(&l1, 0x1007, &model::ints(&[23]), 1); + l1.receive_console(&[0x14]); + l1.receive_console(b"pwr d\r"); + assert!(l1.node_powered()); + assert!(actions.lock().is_empty()); + let mut warning = vec![0x57, 0, 4, 0x10, 4, 0, 0, 0x80, 0, 2]; + warning.extend(model::ints(&[0])); + warning.push(1); + warning.extend_from_slice(b"\r\n001c01 ATTN: Power Down: issue 'pwr d' again to immediately power down.\r\n\0"); + assert_eq!(drain(&l1), encode(&warning)); + l1.power_on(); + l1.receive_console(&[0x14]); + l1.receive_console(b"pwr d\r"); + assert!(l1.node_powered(), "reset clears the first confirmation"); + drain(&l1); + l1.receive_console(&[0x14]); + l1.receive_console(b"power down\r"); + assert!(!l1.node_powered()); + assert_eq!(*actions.lock(), [L1Event::Power(false), L1Event::Shutdown]); +} + +/// Protocol power-down actions must finish before the L1 request reply. +#[test] +fn protocol_and_task_three_power_down_commands_are_immediate() { + let protocol = vec![0x10, 0x0f, 0xff, 0xf0, 6, 0x20, 5, 0]; + let mut task_three = vec![0x10, 0x0f, 0xff, 0xf0, 3, 0, 0, 1, 1]; + task_three.extend_from_slice(b"pwr d\0"); + for (case, packet) in [ + ("protocol command", protocol), + ("task three console", task_three), + ] { + let l1 = L1::new(Default::default()).unwrap(); + let actions = Arc::new(Mutex::new(Vec::new())); + let observed = actions.clone(); + l1.set_event_handler(Arc::new(move |event| observed.lock().push(event))); + send(&l1, &encode(&packet)); + assert_eq!(drain(&l1), integer_reply(&[]), "{case}"); + assert!(!l1.node_powered(), "{case}"); + assert_eq!( + *actions.lock(), + [L1Event::Power(false), L1Event::Shutdown], + "{case}" + ); + } +} + +/// A CPU may read the UART in its interrupt callback while the worker updates the line. +#[test] +fn l1_worker_irq_delivery_stays_ordered_across_reentrant_cpu_reads() { + use std::sync::mpsc; + struct BlockingIrq { + entered: mpsc::SyncSender<()>, + resume: std::sync::Mutex>, + blocked: AtomicBool, + levels: Mutex>, + device: std::sync::Weak, + } + impl crate::irq::InterruptController for BlockingIrq { + fn set_line(&self, _: u32, level: bool) { + if level && !self.blocked.swap(true, Ordering::AcqRel) { + self.entered.send(()).unwrap(); + self.resume.lock().unwrap().recv().unwrap(); + // A machine interrupt handler may inspect the UART immediately. + assert_eq!(self.device.upgrade().unwrap().read64(0x28).data, 0x60); + } + self.levels.lock().push(level); + } + } + let l1 = Arc::new(L1::new(Default::default()).unwrap()); + let backend = Arc::new(crate::dev::serial_host::CiSerialBackend::new()); + let (entered_tx, entered_rx) = mpsc::sync_channel(1); + let (resume_tx, resume_rx) = mpsc::sync_channel(1); + let irq = Arc::new(BlockingIrq { + entered: entered_tx, + resume: std::sync::Mutex::new(resume_rx), + blocked: AtomicBool::new(false), + levels: Mutex::new(Vec::new()), + device: Arc::downgrade(&l1), + }); + l1.set_irq(IrqLine::new(irq.clone(), 0)); + l1.set_serial_backend(backend.clone()); + l1.write64(8, 1); + l1.start(); + backend.push_host(b"x"); + let entered = entered_rx.recv_timeout(std::time::Duration::from_secs(2)); + if entered.is_ok() { + l1.write64(0x10, 2); + } + resume_tx.send(()).unwrap(); + l1.stop(); + entered.unwrap(); + assert_eq!(*irq.levels.lock(), [false, true, false]); +} + +/// Concurrent worker output and platform actions must preserve their order. +#[test] +fn concurrent_l1_outputs_and_actions_are_delivered_in_order() { + use std::sync::mpsc; + struct BlockingBackend { + entered: mpsc::SyncSender<()>, + resume: std::sync::Mutex>, + output: Mutex>, + } + impl SerialBackend for BlockingBackend { + fn send_byte(&self, byte: u8) { + if byte == b'A' { + self.entered.send(()).unwrap(); + self.resume.lock().unwrap().recv().unwrap(); + } + self.output.lock().push(byte); + } + fn recv_byte(&self) -> std::io::Result { + Err(std::io::ErrorKind::WouldBlock.into()) + } + } + let l1 = Arc::new(L1::new(Default::default()).unwrap()); + let (entered_tx, entered_rx) = mpsc::sync_channel(1); + let (resume_tx, resume_rx) = mpsc::sync_channel(1); + let backend = Arc::new(BlockingBackend { + entered: entered_tx, + resume: std::sync::Mutex::new(resume_rx), + output: Mutex::new(Vec::new()), + }); + l1.set_serial_backend(backend.clone()); + let actions = Arc::new(Mutex::new(Vec::new())); + let captured = actions.clone(); + let weak = Arc::downgrade(&l1); + l1.set_event_handler(Arc::new(move |action| { + // Reentrant register reads must not recursively dispatch notifications. + assert_eq!(weak.upgrade().unwrap().read64(0x28).data, 0x60); + captured.lock().push(action); + })); + { + let mut s = l1.state.lock(); + s.output.extend(b"AB"); + s.actions.push(L1Event::Reset); + } + let worker_device = l1.clone(); + let worker = std::thread::spawn(move || worker_device.flush()); + let entered = entered_rx.recv_timeout(std::time::Duration::from_secs(2)); + if entered.is_ok() { + let mut s = l1.state.lock(); + s.output.extend(b"CD"); + s.actions.push(L1Event::Nmi); + drop(s); + l1.flush(); + } + resume_tx.send(()).unwrap(); + worker.join().unwrap(); + entered.unwrap(); + assert_eq!(*backend.output.lock(), b"ABCD"); + assert_eq!(*actions.lock(), [L1Event::Reset, L1Event::Nmi]); +} diff --git a/src/dev/mace.rs b/src/dev/mace.rs index 460036b4b..08988fb78 100644 --- a/src/dev/mace.rs +++ b/src/dev/mace.rs @@ -36,7 +36,7 @@ pub struct Mace { /// and count accesses per offset. pub regs: Stub, /// The PCI host bridge: its registers here, its windows outside MACE. - pub pci: MacePci, + pub pci: Arc, pub ust: Arc, pub rtc: MaceRtc, pub ps2: MacePs2, @@ -55,12 +55,12 @@ impl Mace { /// MACE with its blocks reset, its interrupts wired into `crime`: the /// Ethernet straight in, the ISA peripherals (the PS/2 ports, audio, the /// console UART while driven by PIO, serial DMA) through the ISA block. - pub fn new(ram: Arc, crime: Arc, ust: Arc) -> Self { + pub fn new(ram: Arc, crime: Arc, ust: Arc, pci: &'static crate::pci::IntxRouting) -> Self { let isa = Arc::new(MaceIsa::new(crime.clone())); let mace = Self { isa, regs: Stub::new("mace"), - pci: MacePci::new(crime.clone(), ram.clone()), + pci: MacePci::new(pci, crime.clone(), ram.clone()), ust, rtc: MaceRtc::new(), ps2: MacePs2::new(), @@ -111,7 +111,7 @@ impl Mace { /// commands through on the Indigo2. pub fn devices(&self) -> Vec<&dyn crate::traits::Device> { vec![ - &self.pci, &*self.ust, &self.rtc, &self.ps2, &self.id, &self.com0, &self.com1, &self.sdma, + &*self.pci, &*self.ust, &self.rtc, &self.ps2, &self.id, &self.com0, &self.com1, &self.sdma, &self.audio, &*self.mec, &*self.isa, ] } @@ -276,7 +276,7 @@ impl Mace { [ ("ust", &*self.ust), ("regs", &self.regs), - ("pci", &self.pci), + ("pci", &*self.pci), ("rtc", &self.rtc), ("ps2", &self.ps2), ("id", &self.id), diff --git a/src/dev/mace_pci.rs b/src/dev/mace_pci.rs index ff1f41c91..38003a85c 100644 --- a/src/dev/mace_pci.rs +++ b/src/dev/mace_pci.rs @@ -1,14 +1,15 @@ //! MACE's PCI host bridge on the O2: the root complex. It generates //! configuration cycles, gives the CPU its windows onto PCI memory (swapping -//! byte lanes on the way, since PCI is little-endian and the CPU is not), and -//! wires each slot's interrupt pin to its CRIME input. +//! byte order on the way, since PCI is little-endian and the CPU is not), and +//! passes each slot's interrupt pin to its CRIME input. The functions, their +//! BAR windows and the interrupt combining are the [`PciBus`] it owns. -use std::sync::atomic::{AtomicU64, AtomicUsize, Ordering}; -use std::sync::{Arc, Mutex, OnceLock, RwLock}; +use std::sync::{Arc, Mutex, Weak}; -use crate::irq::{InterruptController, IrqLine}; +use crate::irq::InterruptController; use crate::dev::crime_mem::O2Memory; -use crate::pci::{bar_runs, BarAt, BarRun, BarRunSink, PciBar, PciBusDevice, PciDma, PciDmaStatus, PciLanes, Space}; +use crate::pci::{CpuWindow, DevFn, IntxRouting, PciBus, PciDma, PciDmaStatus, PciHost, Space}; +use crate::pci::ByteOrder; use crate::snapshot::{get_field, hex_u32, load_u32_slice, toml_u32, u32_slice_to_toml}; use crate::traits::{BusDevice, BusRead32, BusRead64, Saveable, BUS_OK}; use crate::ip32::*; @@ -32,65 +33,6 @@ pub const MACEPCI_REVISION_VALUE: u32 = 1; pub const IP32_SCSI_SLOT: u8 = 1; pub const IP32_SCSI_SLOT_B: u8 = 2; -/// Which of CRIME's interrupt inputs the pin of the function in that slot -/// drives. NetBSD works this out for itself and prints it: "interrupting at -/// crime interrupt 8". -pub const IP32_SCSI_CRIME_INT: u32 = 8; -pub const IP32_SCSI_CRIME_INT_B: u32 = 9; - -/// The O2's routing: which CRIME input a slot's pin (1 = INTA) drives. -fn crime_input(slot: u8, pin: u8) -> Option { - match (slot, pin) { - (IP32_SCSI_SLOT, 1) => Some(IP32_SCSI_CRIME_INT), - (IP32_SCSI_SLOT_B, 1) => Some(IP32_SCSI_CRIME_INT_B), - _ => None, - } -} - -/// Bus 0 has 32 device numbers, each with four interrupt pins. -const PCI_PINS: usize = 32 * 4; - -/// The bridge's interrupt router. Each slot's INTA..INTD is a line -/// (`slot * 4 + pin - 1`); the router keeps every pin's level and drives -/// each system input it routes to with the OR of the pins routed there, so -/// on a shared input one function cannot clear another's interrupt. -struct PciIrqRouter { - /// Where the system inputs are: CRIME. - intc: Arc, - /// Slot and pin (1 = INTA) to system input. - route: fn(u8, u8) -> Option, - levels: Mutex<[bool; PCI_PINS]>, -} - -impl PciIrqRouter { - fn line(slot: u8, pin: u8) -> u32 { - slot as u32 * 4 + (pin as u32 - 1) - } - - fn slot_pin(line: u32) -> (u8, u8) { - ((line / 4) as u8, (line % 4) as u8 + 1) - } -} - -impl InterruptController for PciIrqRouter { - fn set_line(&self, line: u32, level: bool) { - if line as usize >= PCI_PINS { - return; - } - let (slot, pin) = Self::slot_pin(line); - let Some(input) = (self.route)(slot, pin) else { return }; - // The lock is held while CRIME is driven, so two pins sharing an - // input cannot reach it out of order. - let mut levels = self.levels.lock().unwrap(); - levels[line as usize] = level; - let any = (0..PCI_PINS as u32).any(|l| { - let (s, p) = Self::slot_pin(l); - levels[l as usize] && (self.route)(s, p) == Some(input) - }); - self.intc.set_line(input, any); - } -} - /// The CPU-side window onto PCI memory, and the PCI address it starts at. /// /// These are not the same number, which is the whole point. The PROM programs @@ -200,167 +142,56 @@ impl PciDma for HostMemory { } } -/// A function in a slot, and the configuration registers the guest has -/// touched (first touch, with direction), for the bring-up report. -struct Slot { - name: &'static str, - dev: Arc, - touched: Mutex>, -} - -/// The most BAR windows open at once (two AIC-7880s open two each). -const MAX_WINDOWS: usize = 16; - -/// A BAR window as the bridge's decode table holds it, packed into one -/// word: base (bits 0-31), log2 of the size (32-37), I/O space (38), slot -/// (40-44), BAR (48-50). -#[derive(Clone, Copy, PartialEq, Eq, Debug)] -struct Window { - space: Space, - slot: u8, - bar: usize, - base: u32, - size: u32, -} - -impl Window { - fn pack(self) -> u64 { - self.base as u64 - | (self.size.trailing_zeros() as u64) << 32 - | ((self.space == Space::Io) as u64) << 38 - | (self.slot as u64) << 40 - | (self.bar as u64) << 48 - } - - fn unpack(w: u64) -> Self { - Self { - space: if w & (1 << 38) != 0 { Space::Io } else { Space::Mem }, - slot: ((w >> 40) & 0x1f) as u8, - bar: ((w >> 48) & 7) as usize, - base: w as u32, - size: 1u32 << ((w >> 32) & 0x3f), - } - } -} - pub struct MacePci { + /// The functions on bus 0, their windows and interrupt pins. + bus: PciBus, + /// Where the bus's outputs go. + crime: Arc, + /// Host memory, for the functions that master the bus. One object: + /// MACE's translation is fixed. + dma: Arc, config_addr: Mutex, regs: Mutex<[u32; (MACEPCI_SIZE / 4) as usize]>, - /// Distinct config addresses the guest selected, in order. - probed: Mutex>, - /// Functions by device number on bus 0. - slots: Mutex>>, - /// The function in each slot, for the CPU's window accesses, which - /// look it up without a lock. - funcs: [OnceLock>; 32], - /// The BAR windows the functions decode, rebuilt from their headers - /// after every configuration write. A function's BAR comparators are - /// its own registers on the bus, not anything the bridge locks, so - /// window accesses read the table without one; configuration writes - /// and window accesses both come from the CPU. - windows: [AtomicU64; MAX_WINDOWS], - nwindows: AtomicUsize, - /// The slots' interrupt pins, routed to CRIME. - irq: Arc, - /// Host memory, for the functions that master the bus. - dma: Arc, - /// Where the runs of the windows a BAR decodes whole go (the machine - /// maps them over MACE's windows), and the entries made for them. - bar_sink: RwLock>, - bar_entries: Mutex>>, } -impl MacePci { - pub fn new(intc: Arc, ram: Arc) -> Self { - Self::with_routing(intc, crime_input, ram) +impl PciHost for MacePci { + fn dma_for(&self, _requester: u8) -> Arc { + self.dma.clone() + } + + fn set_irq(&self, output: u8, level: bool) { + self.crime.set_line(output as u32, level); } +} - /// A bridge with its own routing of slot pins (1 = INTA) to `intc`'s - /// inputs. - fn with_routing(intc: Arc, route: fn(u8, u8) -> Option, ram: Arc) -> Self { - Self { +impl MacePci { + /// A bridge whose bus is wired as `wiring` (its outputs are `crime`'s + /// inputs). + pub fn new(wiring: &'static IntxRouting, crime: Arc, ram: Arc) -> Arc { + let this = Arc::new_cyclic(|me: &Weak| Self { + bus: PciBus::new(0, wiring, me.clone()), + crime, + dma: Arc::new(HostMemory::new(ram)), config_addr: Mutex::new(0), regs: Mutex::new([0u32; (MACEPCI_SIZE / 4) as usize]), - probed: Mutex::new(Vec::new()), - slots: Mutex::new(std::collections::BTreeMap::new()), - funcs: std::array::from_fn(|_| OnceLock::new()), - windows: std::array::from_fn(|_| AtomicU64::new(0)), - nwindows: AtomicUsize::new(0), - irq: Arc::new(PciIrqRouter { intc, route, levels: Mutex::new([false; PCI_PINS]) }), - dma: Arc::new(HostMemory::new(ram)), - bar_sink: RwLock::new(None), - bar_entries: Mutex::new(Default::default()), - } + }); + // The windows are fixed: MACE has no register that moves them. + this.bus.set_host_windows(&Self::windows); + this } - /// Put a function in a slot on bus 0, wire its interrupt pin to the - /// bridge, which routes it on, and give it host memory to master. - pub fn attach(&self, slot: u8, name: &'static str, dev: Arc) { - dev.connect_dma(self.dma.clone()); - let pin = dev.config().interrupt_pin(); - if (1..=4).contains(&pin) { - dev.connect_interrupt(IrqLine::new(self.irq.clone(), PciIrqRouter::line(slot, pin))); - } - assert!(self.funcs[slot as usize].set(dev.clone()).is_ok(), "pci: slot {slot} is taken"); - let mut slots = self.slots.lock().unwrap(); - slots.insert(slot, Arc::new(Slot { name, dev, touched: Mutex::new(Vec::new()) })); - self.rebuild_windows(&slots); - } - - /// Rebuild the decode table from the functions' headers, slot by slot - /// and BAR by BAR, the order in which an overlap resolves. - fn rebuild_windows(&self, slots: &std::collections::BTreeMap>) { - let mut n = 0; - for (&slot, s) in slots { - for (space, bar, base, size) in s.dev.config().windows() { - if n == MAX_WINDOWS { - log::warn!("pci: more than {MAX_WINDOWS} BAR windows open; slot {slot} BAR {bar} ignored"); - continue; - } - self.windows[n].store(Window { space, slot, bar, base, size }.pack(), Ordering::Relaxed); - n += 1; - } - } - self.nwindows.store(n, Ordering::Release); - self.publish_bars(slots); - } - - /// Send the runs of the CPU's PCI windows that one BAR decodes whole to - /// `sink`, now and after every configuration write. Each run is a - /// [`PciBar`] at its CPU address: the machine maps it on the physical - /// bus over MACE, so the processor reaches the function with no search - /// of the windows. Everything else in a window (I/O BARs smaller than a - /// page, unclaimed addresses, which float high) still comes here. - pub fn on_bar_runs(&self, sink: BarRunSink) { - *self.bar_sink.write().unwrap() = Some(sink); - self.publish_bars(&self.slots.lock().unwrap()); - } - - fn publish_bars(&self, slots: &std::collections::BTreeMap>) { - let sink = self.bar_sink.read().unwrap(); - let Some(sink) = sink.as_ref() else { return }; - let funcs: Vec<(usize, Arc)> = slots.iter().map(|(&i, s)| (i as usize, s.dev.clone())).collect(); - let mut entry = |b: BarAt| { - self.bar_entries - .lock() - .unwrap() - .entry((b.slot, b.bar, b.cpu_base)) - .or_insert_with(|| Arc::new(PciBar::new(Arc::new(Lanes), b.dev.clone(), b.bar, 0xffff_ffff, b.cpu_base))) - .clone() - }; - let mut runs: Vec = Vec::new(); - runs.extend(bar_runs(&funcs, Space::Io, PCI_IO_BASE as u64, 0, PCI_IO_SIZE as u64, &mut entry)); - runs.extend(bar_runs(&funcs, Space::Mem, PCI_MEM_BASE as u64, PCI_MEM_PCI_BASE, PCI_MEM_SIZE as u64, &mut entry)); - let hi = (1u64 << 32) - PCI_HI_MEM_BASE as u64; - runs.extend(bar_runs(&funcs, Space::Mem, PCI_HI_MEM_BASE as u64, PCI_HI_MEM_BASE, hi, &mut entry)); - sink(runs); + pub fn bus(&self) -> &PciBus { + &self.bus } - /// The function in a slot: its name and the registers the guest touched. - pub fn slot(&self, slot: u8) -> Option<(&'static str, Arc, Vec<(u8, bool)>)> { - let s = self.slots.lock().unwrap().get(&slot).cloned()?; - let touched = s.touched.lock().unwrap().clone(); - Some((s.name, s.dev.clone(), touched)) + /// The CPU's three windows onto PCI: I/O, memory low and memory high. + fn windows() -> Vec { + let w = |space, cpu: u32, pci: u32, len: u64| CpuWindow { space, cpu: cpu as u64, pci, len, mask: 0xffff_ffff }; + vec![ + w(Space::Io, PCI_IO_BASE, 0, PCI_IO_SIZE as u64), + w(Space::Mem, PCI_MEM_BASE, PCI_MEM_PCI_BASE, PCI_MEM_SIZE as u64), + w(Space::Mem, PCI_HI_MEM_BASE, PCI_HI_MEM_BASE, (1u64 << 32) - PCI_HI_MEM_BASE as u64), + ] } /// Whether `addr` is in one of the CPU's windows onto PCI memory. @@ -382,136 +213,53 @@ impl MacePci { PCI_MEM_PCI_BASE.wrapping_add(addr.wrapping_sub(PCI_MEM_BASE)) } - /// The bridge swaps byte lanes: an 8-bit access at CPU offset `A` reaches - /// device byte `A ^ 3`, and a 16-bit one `A ^ 2`. - /// - /// This is not cosmetic. Without it the AIC-7880's initialisation reads - /// as writes to registers 0x84, 0x91, 0x93 and 0xbc, which are nothing in - /// particular. With it they are HCNTRL (chip reset, then pause), CLRINT - /// (clear every interrupt), SCBPTR walking 0,1,2,... and the last byte of - /// each SCB being cleared — which is exactly how that chip is brought up. - pub fn lane_swizzle(off: u32, width: usize) -> u32 { - match width { - 1 => off ^ 3, - 2 => off ^ 2, - _ => off, - } - } - - /// The function decoding a PCI memory address: it, the BAR and the - /// offset within it. - fn mem_target(&self, space: Space, pci_addr: u32) -> Option<(&Arc, usize, u32)> { - let n = self.nwindows.load(Ordering::Acquire); - self.windows[..n].iter().find_map(|w| { - let w = Window::unpack(w.load(Ordering::Relaxed)); - let off = pci_addr.wrapping_sub(w.base); - if w.space != space || off >= w.size { - return None; - } - Some((self.funcs[w.slot as usize].get()?, w.bar, off)) - }) + /// The bridge swaps byte order: an 8-bit access at CPU offset `A` reaches + /// device byte `A ^ 3`, and a 16-bit one `A ^ 2`. Without it the + /// AIC-7880's initialisation reads as writes to registers that are + /// nothing in particular; with it they are HCNTRL (chip reset, then + /// pause), CLRINT, SCBPTR walking 0,1,2,... and the last byte of each + /// SCB being cleared, which is how that chip is brought up. + fn order() -> ByteOrder { + ByteOrder::values(4) } - /// The function and BAR a CPU address in a window reaches, in whichever - /// space that window opens onto. - fn window_target(&self, addr: u32) -> Option<(&Arc, usize, u32)> { - if Self::in_io_window(addr) { - return self.mem_target(Space::Io, addr - PCI_IO_BASE); - } - self.mem_target(Space::Mem, Self::pci_addr_of(addr)) - } - - fn mem_read(&self, addr: u32, width: usize) -> u32 { - match self.window_target(addr) { - Some((f, bar, off)) => f.bar_read(bar, Self::lane_swizzle(off, width), width), - // Nobody claims it: the bus floats high. - None => 0xffff_ffff, - } - } - - fn mem_write(&self, addr: u32, val: u32, width: usize) { - if let Some((f, bar, off)) = self.window_target(addr) { - f.bar_write(bar, Self::lane_swizzle(off, width), val, width); - } - } - - /// A CPU read of `width` bytes in a PCI memory window. PCI is 32 bits - /// wide, so a doubleword is two cycles, the high word first. + /// A CPU read of `width` bytes in a PCI window. A doubleword is two + /// word cycles, the high word first. Nobody claims it: the bus floats + /// high. pub fn window_read(&self, addr: u32, width: usize) -> u64 { - if width == 8 { - let hi = self.mem_read(addr, 4) as u64; - return (hi << 32) | self.mem_read(addr.wrapping_add(4), 4) as u64; + let target = if Self::in_io_window(addr) { + self.bus.decode(Space::Io, addr - PCI_IO_BASE) + } else { + self.bus.decode(Space::Mem, Self::pci_addr_of(addr)) + }; + match target { + Some((f, bar, off)) => Self::order().read(f, bar, off, width), + None if width == 8 => u64::MAX, + None => 0xffff_ffff, } - self.mem_read(addr, width) as u64 } pub fn window_write(&self, addr: u32, val: u64, width: usize) { - if width == 8 { - self.mem_write(addr, (val >> 32) as u32, 4); - self.mem_write(addr.wrapping_add(4), val as u32, 4); - return; + let target = if Self::in_io_window(addr) { + self.bus.decode(Space::Io, addr - PCI_IO_BASE) + } else { + self.bus.decode(Space::Mem, Self::pci_addr_of(addr)) + }; + if let Some((f, bar, off)) = target { + Self::order().write(f, bar, off, width, val); } - self.mem_write(addr, val as u32, width); } - /// Split a CONFIG_ADDR value into the function it selects and the - /// register within it. Only bus 0 exists here. - fn selected(&self) -> Option<(Arc, usize)> { - // Bit 31 is the enable bit of the standard PCI mechanism, and MACE's - // bridge does not insist on it: the PROM sets it (0x80000800) and - // NetBSD's driver does not (0x00000800). Requiring it makes the whole - // bus invisible to one of them, which is exactly how the kernel came - // to probe and find nothing. + /// The bus, function and byte offset CONFIG_ADDR selects. Bit 31 is the + /// enable bit of the standard PCI mechanism, and MACE's bridge does not + /// insist on it: the PROM sets it (0x80000800) and NetBSD's driver does + /// not (0x00000800). Requiring it makes the whole bus invisible to one + /// of them. Bus 0 is MACE's own; another number is a bus behind a + /// PCI-to-PCI bridge, if firmware has numbered one that far. + fn selected(&self) -> Option<(PciBus, DevFn, usize)> { let a = *self.config_addr.lock().unwrap() & 0x7fff_ffff; - if (a >> 16) & 0xff != 0 { - return None; - } - let dev = ((a >> 11) & 0x1f) as u8; - let func = (a >> 8) & 7; - if func != 0 { - return None; - } - let s = self.slots.lock().unwrap().get(&dev).cloned()?; - Some((s, ((a >> 2) & 0x3f) as usize)) - } - - /// Config addresses selected so far, in first-touch order. - pub fn probed(&self) -> Vec { - self.probed.lock().unwrap().clone() - } -} - -impl Slot { - fn note(&self, reg: usize, write: bool) { - let mut t = self.touched.lock().unwrap(); - let r = (reg * 4) as u8; - if !t.contains(&(r, write)) { - t.push((r, write)); - } - } -} - -/// MACE's lanes, for [`PciBar`] entries: the same swizzle and the same -/// split of a doubleword into two cycles, high word first, that -/// `window_read` and `window_write` make once they have found the target. -struct Lanes; - -impl PciLanes for Lanes { - fn read(&self, f: &dyn PciBusDevice, bar: usize, off: u32, size: usize) -> u64 { - if size == 8 { - let hi = f.bar_read(bar, off, 4) as u64; - return hi << 32 | f.bar_read(bar, off.wrapping_add(4), 4) as u64; - } - f.bar_read(bar, MacePci::lane_swizzle(off, size), size) as u64 - } - - fn write(&self, f: &dyn PciBusDevice, bar: usize, off: u32, size: usize, val: u64) { - if size == 8 { - f.bar_write(bar, off, (val >> 32) as u32, 4); - f.bar_write(bar, off.wrapping_add(4), val as u32, 4); - return; - } - f.bar_write(bar, MacePci::lane_swizzle(off, size), val as u32, size); + let bus = self.bus.find_bus(((a >> 16) & 0xff) as u8)?; + Some((bus, DevFn { device: ((a >> 11) & 0x1f) as u8, function: ((a >> 8) & 7) as u8 }, (a & 0xfc) as usize)) } } @@ -520,16 +268,12 @@ impl BusDevice for MacePci { fn read32(&self, addr: u64) -> BusRead32 { let off = (addr as u32) & (MACEPCI_SIZE - 1) & !3; match off { - macepci_reg::CONFIG_DATA => BusRead32::ok(match self.selected() { - Some((s, reg)) => { - s.note(reg, false); - s.dev.config().read(reg) - } - // An absent device: the bus floats high. Zero would read back - // as vendor ID 0x0000, which firmware takes for a device that - // is present but broken. - None => 0xffff_ffff, - }), + // An absent device: the bus floats high. Zero would read back + // as vendor ID 0x0000, which firmware takes for a device that + // is present but broken. + macepci_reg::CONFIG_DATA => BusRead32::ok( + self.selected().and_then(|(bus, at, off)| bus.config_read(at, off, 4)).unwrap_or(0xffff_ffff), + ), macepci_reg::CONFIG_ADDR => BusRead32::ok(*self.config_addr.lock().unwrap()), macepci_reg::REVISION => BusRead32::ok(MACEPCI_REVISION_VALUE), _ => BusRead32::ok(self.regs.lock().unwrap()[(off / 4) as usize]), @@ -539,18 +283,10 @@ impl BusDevice for MacePci { fn write32(&self, addr: u64, val: u32) -> u32 { let off = (addr as u32) & (MACEPCI_SIZE - 1) & !3; match off { - macepci_reg::CONFIG_ADDR => { - *self.config_addr.lock().unwrap() = val; - let mut p = self.probed.lock().unwrap(); - if val != 0 && !p.contains(&val) { - p.push(val); - } - } + macepci_reg::CONFIG_ADDR => *self.config_addr.lock().unwrap() = val, macepci_reg::CONFIG_DATA => { - if let Some((s, reg)) = self.selected() { - s.note(reg, true); - s.dev.config().write(reg, val); - self.rebuild_windows(&self.slots.lock().unwrap()); + if let Some((bus, at, off)) = self.selected() { + bus.config_write(at, off, 4, val); } } _ => self.regs.lock().unwrap()[(off / 4) as usize] = val, @@ -570,45 +306,30 @@ impl BusDevice for MacePci { } } -/// The bridge's registers, the selected configuration address, the pins' -/// levels (driven into CRIME again on load) and each slot's configuration -/// header, by device number. The functions behind the slots save their own -/// state. +/// The bridge's registers, the selected configuration address and the bus +/// (each function's header, the pin and wire levels). The functions behind +/// the headers save their own state; the machine drives the pins into CRIME +/// again with [`PciBus::redrive`] once CRIME has loaded. impl Saveable for MacePci { fn save_state(&self) -> toml::Value { let mut t = toml::map::Map::new(); t.insert("config_addr".into(), hex_u32(*self.config_addr.lock().unwrap())); t.insert("regs".into(), u32_slice_to_toml(&*self.regs.lock().unwrap())); - let levels = *self.irq.levels.lock().unwrap(); - t.insert("pins".into(), toml::Value::Array(levels.iter().map(|&b| toml::Value::Boolean(b)).collect())); - let mut slots = toml::map::Map::new(); - for (n, s) in self.slots.lock().unwrap().iter() { - slots.insert(n.to_string(), s.dev.config().save_state()); - } - t.insert("slots".into(), toml::Value::Table(slots)); + t.insert("bus".into(), self.bus.save_state()); toml::Value::Table(t) } fn load_state(&self, v: &toml::Value) -> Result<(), String> { *self.config_addr.lock().unwrap() = get_field(v, "config_addr").and_then(toml_u32).unwrap_or(0); + // A field the snapshot lacks is the power-on value, not what the + // machine held before the load. + let mut regs = self.regs.lock().unwrap(); + *regs = [0; (MACEPCI_SIZE / 4) as usize]; if let Some(r) = get_field(v, "regs") { - load_u32_slice(r, &mut *self.regs.lock().unwrap()); + load_u32_slice(r, &mut *regs); } - if let Some(toml::Value::Table(slots)) = get_field(v, "slots") { - let ours = self.slots.lock().unwrap(); - for (n, cfg) in slots { - let s = n.parse::().ok().and_then(|n| ours.get(&n)) - .ok_or_else(|| format!("pci: saved state has slot {n}, which is empty here"))?; - s.dev.config().load_state(cfg)?; - } - self.rebuild_windows(&ours); - } - if let Some(toml::Value::Array(pins)) = get_field(v, "pins") { - for (line, level) in pins.iter().enumerate().take(PCI_PINS) { - self.irq.set_line(line as u32, level.as_bool().unwrap_or(false)); - } - } - Ok(()) + drop(regs); + self.bus.load_state(get_field(v, "bus").ok_or("pci: saved MACE state has no bus")?) } } @@ -635,30 +356,9 @@ impl crate::traits::Device for MacePci { writeln!(w, "MACE PCI: control {:#010x} revision {:#x} error flags {:#x} at {:#010x}, config address {:#010x}", reg(macepci_reg::CONTROL), reg(macepci_reg::REVISION), reg(macepci_reg::ERROR_FLAGS), reg(macepci_reg::ERROR_ADDR), *self.config_addr.lock().unwrap()).ok(); - for (slot, s) in self.slots.lock().unwrap().iter() { - let c = s.dev.config(); - let id = c.read(0); - let bars: Vec = (0..6).map(|i| c.bar_base(i)).filter(|&b| b != 0).map(|b| format!("{b:#x}")).collect(); - let irq = match c.interrupt_pin() { - 0 => "no interrupt".to_string(), - p => match (self.irq.route)(*slot, p) { - Some(l) => format!("INT{} to CRIME {l}", (b'A' + p - 1) as char), - None => format!("INT{} unrouted", (b'A' + p - 1) as char), - }, - }; - writeln!(w, " slot {slot}: {} {:04x}:{:04x}, command {:#06x}, BARs [{}], {irq}", - s.name, id & 0xffff, id >> 16, c.command(), bars.join(", ")).ok(); - } - } - "config" => { - let slot: u8 = args.get(1).and_then(|v| v.parse().ok()).ok_or("Usage: pci config ")?; - let (_, f, _) = self.slot(slot).ok_or(format!("pci: nothing in slot {slot}"))?; - for row in 0..16 { - let v: Vec = (0..4).map(|k| format!("{:08x}", f.config().read(row * 4 + k))).collect(); - writeln!(w, " {:02x}: {}", row * 16, v.join(" ")).ok(); - } + self.bus.describe(&mut w); } - _ => return Err("Usage: pci status | config ".to_string()), + _ => return self.bus.command(args, &mut w), } Ok(()) } @@ -667,26 +367,9 @@ impl crate::traits::Device for MacePci { #[cfg(test)] mod tests { use super::*; - use crate::pci::{Bar, PciConfig, PCI_COMMAND_IO, PCI_COMMAND_MEMORY, PCI_INTA}; - - /// A function that remembers its last BAR access and drives its pin. - struct Fake { - cfg: PciConfig, - last: Mutex>, - irq: std::sync::OnceLock, - } - - impl PciBusDevice for Fake { - fn config(&self) -> &PciConfig { &self.cfg } - fn bar_read(&self, bar: usize, off: u32, width: usize) -> u32 { - *self.last.lock().unwrap() = Some((bar, off, width)); - 0x1234_5678 - } - fn bar_write(&self, bar: usize, off: u32, _val: u32, width: usize) { - *self.last.lock().unwrap() = Some((bar, off, width)); - } - fn connect_interrupt(&self, line: IrqLine) { let _ = self.irq.set(line); } - } + use crate::pci::BarMapping; + use crate::pci::testing::TestCard; + use crate::pci::{Bar, PciFunction, PCI_COMMAND_IO, PCI_COMMAND_MEMORY, PCI_INTA}; struct Crime(Mutex>); impl InterruptController for Crime { @@ -701,6 +384,18 @@ mod tests { Arc::new(O2Memory::single(1 << 20)) } + fn bars() -> [Bar; 6] { + [Bar::Io(0x100), Bar::Mem(0x1000), Bar::Unused, Bar::Unused, Bar::Unused, Bar::Unused] + } + + fn bridge_with(crime: &Arc) -> Arc { + MacePci::new(&PCI0, crime.clone(), test_ram()) + } + + fn at(device: u8) -> DevFn { + DevFn { device, function: 0 } + } + /// A bus master reaches the same memory through both views, bytes in /// address order, and nothing above 2 GB. #[test] @@ -726,59 +421,49 @@ mod tests { #[test] fn the_bridge_configures_decodes_and_routes_a_function() { let crime = Arc::new(Crime(Mutex::new(Vec::new()))); - let pci = MacePci::new(crime.clone(), test_ram()); - let f = Arc::new(Fake { - cfg: PciConfig::new(0x9004, 0x8078, 0x0100_00, 1, - [Bar::Io(0x100), Bar::Mem(0x1000), Bar::Unused, Bar::Unused, Bar::Unused, Bar::Unused], PCI_INTA), - last: Mutex::new(None), - irq: std::sync::OnceLock::new(), - }); - pci.attach(IP32_SCSI_SLOT, "fake", f.clone()); + let pci = bridge_with(&crime); + let f = TestCard::new(bars(), PCI_INTA); + pci.bus().attach(at(IP32_SCSI_SLOT), f.clone()); // Config mechanism #1: slot 1 answers, slot 3 floats high. let slot1 = (IP32_SCSI_SLOT as u32) << 11; pci.write32(bridge(macepci_reg::CONFIG_ADDR), cfg(slot1, 0)); - assert_eq!(pci.read32(bridge(macepci_reg::CONFIG_DATA)).data, 0x8078_9004); + assert_eq!(pci.read32(bridge(macepci_reg::CONFIG_DATA)).data, 0x5678_1234); pci.write32(bridge(macepci_reg::CONFIG_ADDR), cfg(3 << 11, 0)); assert_eq!(pci.read32(bridge(macepci_reg::CONFIG_DATA)).data, 0xffff_ffff); // Assign BAR1 and enable memory: the low window reaches it, with - // byte lanes swapped. + // byte order swapped. pci.write32(bridge(macepci_reg::CONFIG_ADDR), cfg(slot1, 5)); pci.write32(bridge(macepci_reg::CONFIG_DATA), 0x8000_1000); assert_eq!(pci.window_read(PCI_MEM_BASE + 0x1084, 4), 0xffff_ffff, "not before memory is enabled"); pci.write32(bridge(macepci_reg::CONFIG_ADDR), cfg(slot1, 1)); pci.write32(bridge(macepci_reg::CONFIG_DATA), PCI_COMMAND_MEMORY); - assert_eq!(pci.window_read(PCI_MEM_BASE + 0x1084, 4), 0x1234_5678); - assert_eq!(*f.last.lock().unwrap(), Some((1, 0x84, 4))); + pci.window_read(PCI_MEM_BASE + 0x1084, 4); + assert_eq!(f.calls(), vec![(0x84, 4)]); pci.window_write(PCI_MEM_BASE + 0x1084, 0, 1); - assert_eq!(*f.last.lock().unwrap(), Some((1, 0x87, 1)), "HCNTRL, by its little-endian lane"); - assert_eq!(pci.window_read(0x8000_1084, 2) as u16, 0x5678, "and the one-to-one high window"); - assert_eq!(*f.last.lock().unwrap(), Some((1, 0x86, 2))); - assert_eq!(pci.slot(IP32_SCSI_SLOT).unwrap().2, vec![(0x00, false), (0x14, true), (0x04, true)], - "config registers touched, for the bring-up report"); + assert_eq!(f.calls()[1], (0x87, 1), "HCNTRL, by its little-endian lane"); + pci.window_read(0x8000_1084, 2); + assert_eq!(f.calls()[2], (0x86, 2), "and the one-to-one high window"); // Pin A of slot 1 is CRIME input 8. - f.irq.get().expect("wired at attach").set(true); - assert_eq!(*crime.0.lock().unwrap(), vec![(IP32_SCSI_CRIME_INT, true)]); + f.state().irq().expect("wired at attach").set(true); + assert_eq!(*crime.0.lock().unwrap(), vec![(8, true)]); } + /// Once a memory BAR is placed and enabled, the pages it decodes in both /// CPU windows are published as runs, and an access through a run is - /// the window's access, lanes included. + /// the window's access, order included. #[test] fn a_placed_bar_is_published_as_runs() { + use crate::pci::BarDevice; let crime = Arc::new(Crime(Mutex::new(Vec::new()))); - let pci = MacePci::new(crime, test_ram()); - let f = Arc::new(Fake { - cfg: PciConfig::new(0x9004, 0x8078, 0x0100_00, 1, - [Bar::Io(0x100), Bar::Mem(0x1000), Bar::Unused, Bar::Unused, Bar::Unused, Bar::Unused], PCI_INTA), - last: Mutex::new(None), - irq: std::sync::OnceLock::new(), - }); - pci.attach(IP32_SCSI_SLOT, "fake", f.clone()); - let runs = Arc::new(Mutex::new(Vec::::new())); + let pci = bridge_with(&crime); + let f = TestCard::new(bars(), PCI_INTA); + pci.bus().attach(at(IP32_SCSI_SLOT), f.clone()); + let runs = Arc::new(Mutex::new(Vec::::new())); let sink = runs.clone(); - pci.on_bar_runs(Box::new(move |r| *sink.lock().unwrap() = r)); + pci.bus().on_bar_mappings(Box::new(move |r| *sink.lock().unwrap() = r)); assert!(runs.lock().unwrap().is_empty()); let slot1 = (IP32_SCSI_SLOT as u32) << 11; pci.write32(bridge(macepci_reg::CONFIG_ADDR), cfg(slot1, 5)); @@ -787,77 +472,135 @@ mod tests { pci.write32(bridge(macepci_reg::CONFIG_ADDR), cfg(slot1, 1)); pci.write32(bridge(macepci_reg::CONFIG_DATA), PCI_COMMAND_IO | PCI_COMMAND_MEMORY); let got = runs.lock().unwrap().clone(); - let spans: Vec<_> = got.iter().map(|r| (r.start, r.end)).collect(); + let spans: Vec<_> = got.iter().map(|r| (r.cpu, r.cpu + r.len)).collect(); let low = (PCI_MEM_BASE + 0x1000) as u64; assert_eq!(spans, vec![(low, low + 0x1000), (0x8000_1000, 0x8000_2000)], "the I/O BAR is under a page"); - assert_eq!(got[0].dev.read32(low + 0x84).data, 0x1234_5678); - assert_eq!(*f.last.lock().unwrap(), Some((1, 0x84, 4))); - assert_eq!(got[0].dev.write8(low + 0x84, 0), BUS_OK); - assert_eq!(*f.last.lock().unwrap(), Some((1, 0x87, 1)), "HCNTRL, by its little-endian lane"); - assert_eq!(got[1].dev.read16(0x8000_1084).data, 0x5678); - assert_eq!(*f.last.lock().unwrap(), Some((1, 0x86, 2))); + // The machine builds a BarDevice from each run. + let bar = |r: &BarMapping| BarDevice::new(r.function.clone(), r.bar, r.cpu.wrapping_sub(r.off as u64), 0xffff_ffff, ByteOrder::values(4)); + assert_eq!(bar(&got[0]).read32(low + 0x84).data, 0); + assert_eq!(f.calls(), vec![(0x84, 4)]); + assert_eq!(bar(&got[0]).write8(low + 0x84, 0), BUS_OK); + assert_eq!(f.calls()[1], (0x87, 1), "HCNTRL, by its little-endian lane"); + bar(&got[1]).read16(0x8000_1084); + assert_eq!(f.calls()[2], (0x86, 2)); } #[test] fn save_load_round_trip() { - let make = |crime: Arc| { - let pci = MacePci::new(crime, test_ram()); - let f = Arc::new(Fake { - cfg: PciConfig::new(0x9004, 0x8078, 0x0100_00, 1, - [Bar::Io(0x100), Bar::Mem(0x1000), Bar::Unused, Bar::Unused, Bar::Unused, Bar::Unused], PCI_INTA), - last: Mutex::new(None), - irq: std::sync::OnceLock::new(), - }); - pci.attach(IP32_SCSI_SLOT, "fake", f.clone()); + let make = |crime: &Arc| { + let pci = bridge_with(crime); + let f = TestCard::new(bars(), PCI_INTA); + pci.bus().attach(at(IP32_SCSI_SLOT), f.clone()); (pci, f) }; - let (src, f) = make(Arc::new(Crime(Mutex::new(Vec::new())))); + let (src, f) = make(&Arc::new(Crime(Mutex::new(Vec::new())))); let slot1 = (IP32_SCSI_SLOT as u32) << 11; src.write32(bridge(macepci_reg::CONFIG_ADDR), cfg(slot1, 5)); src.write32(bridge(macepci_reg::CONFIG_DATA), 0x8000_1000); src.write32(bridge(macepci_reg::CONFIG_ADDR), cfg(slot1, 1)); src.write32(bridge(macepci_reg::CONFIG_DATA), PCI_COMMAND_MEMORY); - f.irq.get().unwrap().set(true); + src.write32(bridge(macepci_reg::CONTROL), 0x1234); + f.state().irq().unwrap().set(true); let v1 = src.save_state(); let crime = Arc::new(Crime(Mutex::new(Vec::new()))); - let (dst, _) = make(crime.clone()); + let (dst, g) = make(&crime); dst.load_state(&v1).expect("load_state"); assert_eq!(v1, dst.save_state(), "MacePci save_state mismatch after load_state round-trip"); - assert_eq!(dst.window_read(PCI_MEM_BASE + 0x1084, 4), 0x1234_5678, "BAR and memory enable restored"); - assert!(crime.0.lock().unwrap().contains(&(IP32_SCSI_CRIME_INT, true)), "the pin is driven again"); + dst.window_read(PCI_MEM_BASE + 0x1084, 4); + assert_eq!(g.calls(), vec![(0x84, 4)], "BAR and memory enable restored"); + assert!(crime.0.lock().unwrap().is_empty(), "load drives nothing; the machine redrives"); + dst.bus().redrive(); + assert!(crime.0.lock().unwrap().contains(&(8, true)), "the pin is driven again"); } - /// Two functions whose pins share one system input (slot 3's INTA and - /// slot 4's INTB, on a made-up routing): the input stays up until both - /// have dropped, so neither can clear the other's interrupt. + /// A snapshot lacking a field leaves the power-on value, not whatever + /// the machine held before the load. #[test] - fn a_shared_input_stays_up_until_every_pin_drops() { - fn route(slot: u8, pin: u8) -> Option { - match (slot, pin) { - (3, 1) | (4, 2) => Some(12), - _ => None, - } + fn mace_snapshot_defaults_an_absent_regs_field() { + let crime = Arc::new(Crime(Mutex::new(Vec::new()))); + let pci = bridge_with(&crime); + let mut v = pci.save_state(); + pci.write32(bridge(macepci_reg::CONTROL), 0xdead_beef); + v.as_table_mut().unwrap().remove("regs"); + pci.load_state(&v).unwrap(); + assert_eq!(pci.read32(bridge(macepci_reg::CONTROL)).data, 0); + } + + /// MACE spec Tables 85, 87 and 90: devices 1 and 2 INTA to CRIME 8 and 9, + /// device 3 INTA to 10 and INTB-D to 13 to 15. Device 4 INTA to 11 is + /// iris's spare for virtio-serial. Nothing else is wired. + #[test] + fn o2_routes_devices_1_2_3_4_to_crime() { + let routes = [(1u8, 1u8, 8u32), (2, 1, 9), (3, 1, 10), (3, 2, 13), (3, 3, 14), (3, 4, 15), (4, 1, 11)]; + for (device, pin, input) in routes { + let crime = Arc::new(Crime(Mutex::new(Vec::new()))); + let pci = bridge_with(&crime); + let c = TestCard::new(bars(), pin); + pci.bus().attach(at(device), c.clone()); + c.state().irq().unwrap().set(true); + assert_eq!(*crime.0.lock().unwrap(), vec![(input, true)], "device {device} INT{}", (b'A' + pin - 1) as char); } + } + + #[test] + #[should_panic(expected = "does not route")] + fn a_pin_on_device_5_panics_at_attach() { let crime = Arc::new(Crime(Mutex::new(Vec::new()))); - let pci = MacePci::with_routing(crime.clone(), route, test_ram()); - let fake = |pin| Arc::new(Fake { - cfg: PciConfig::new(0x1234, 0x5678, 0, 0, [Bar::Unused; 6], pin), - last: Mutex::new(None), - irq: std::sync::OnceLock::new(), - }); - let (a, b) = (fake(PCI_INTA), fake(PCI_INTA + 1)); - pci.attach(3, "a", a.clone()); - pci.attach(4, "b", b.clone()); - let (la, lb) = (a.irq.get().unwrap(), b.irq.get().unwrap()); - let last = || *crime.0.lock().unwrap().last().unwrap(); - - la.set(true); - assert_eq!(last(), (12, true)); - lb.set(true); - la.set(false); - assert_eq!(last(), (12, true), "slot 4's INTB still holds it"); - lb.set(false); - assert_eq!(last(), (12, false)); + bridge_with(&crime).bus().attach(at(5), TestCard::new(bars(), PCI_INTA)); + } + + #[test] + fn an_absent_function_floats_high_on_config_read() { + let crime = Arc::new(Crime(Mutex::new(Vec::new()))); + let pci = bridge_with(&crime); + pci.bus().attach(at(1), TestCard::new(bars(), PCI_INTA)); + for sel in [cfg(2 << 11, 0), cfg(1 << 11 | 1 << 8, 0), cfg(1 << 11 | 1 << 16, 0)] { + pci.write32(bridge(macepci_reg::CONFIG_ADDR), sel); + assert_eq!(pci.read32(bridge(macepci_reg::CONFIG_DATA)).data, 0xffff_ffff, "{sel:#x}"); + } + } + + /// Bytes `^3`, halves `^2`, words as given, and a doubleword as two word + /// cycles, high word first (MACE's 64-bit PIO is unknown, see ByteOrder). + #[test] + fn mace_window_read_uses_values_4() { + let crime = Arc::new(Crime(Mutex::new(Vec::new()))); + let pci = bridge_with(&crime); + let f = TestCard::new(bars(), PCI_INTA); + pci.bus().attach(at(1), f.clone()); + pci.write32(bridge(macepci_reg::CONFIG_ADDR), cfg(1 << 11, 5)); + pci.write32(bridge(macepci_reg::CONFIG_DATA), 0x8000_1000); + pci.write32(bridge(macepci_reg::CONFIG_ADDR), cfg(1 << 11, 1)); + pci.write32(bridge(macepci_reg::CONFIG_DATA), PCI_COMMAND_MEMORY); + let base = PCI_MEM_BASE + 0x1000; + pci.window_read(base + 4, 1); + pci.window_read(base + 4, 2); + pci.window_read(base + 4, 4); + pci.window_read(base + 8, 8); + assert_eq!(f.calls(), vec![(7, 1), (6, 2), (4, 4), (8, 4), (12, 4)]); + } + + /// A configuration cycle naming bus 1 reaches the bus behind a + /// PCI-to-PCI bridge once firmware has numbered it, and floats high + /// before. + #[test] + fn a_host_config_cycle_for_bus_1_reaches_the_secondary() { + use crate::pci::PciToPciBridge; + let crime = Arc::new(Crime(Mutex::new(Vec::new()))); + let pci = bridge_with(&crime); + let p2p = PciToPciBridge::new(pci.bus(), at(4), 0x1234, 0x5678); + pci.bus().attach(at(4), p2p.clone()); + p2p.secondary().attach(at(2), TestCard::new(bars(), 0)); + let behind = |reg| cfg(1 << 16 | 2 << 11, reg); + pci.write32(bridge(macepci_reg::CONFIG_ADDR), behind(0)); + assert_eq!(pci.read32(bridge(macepci_reg::CONFIG_DATA)).data, 0xffff_ffff, "no bus 1 yet"); + // Secondary 1, subordinate 1, as the PROM numbers it. + pci.write32(bridge(macepci_reg::CONFIG_ADDR), cfg(4 << 11, 6)); + pci.write32(bridge(macepci_reg::CONFIG_DATA), 0x0001_0100); + pci.write32(bridge(macepci_reg::CONFIG_ADDR), behind(0)); + assert_eq!(pci.read32(bridge(macepci_reg::CONFIG_DATA)).data, 0x5678_1234); + pci.write32(bridge(macepci_reg::CONFIG_ADDR), cfg(2 << 16 | 2 << 11, 0)); + assert_eq!(pci.read32(bridge(macepci_reg::CONFIG_DATA)).data, 0xffff_ffff, "bus 2 is not behind it"); } } diff --git a/src/dev/mod.rs b/src/dev/mod.rs index 99eba9d72..2bace5a5b 100644 --- a/src/dev/mod.rs +++ b/src/dev/mod.rs @@ -61,6 +61,7 @@ pub mod seeq8003; pub mod testdev; pub mod timer; pub mod ultra64; +pub mod usb_hid; pub mod vino; pub mod virtio; pub mod wd33c93a; @@ -72,7 +73,19 @@ pub mod hub_md; pub mod hub_ni; pub mod hub_pi; pub mod dp83840; +pub mod bedrock_flash; +pub mod bedrock_io; +pub mod bedrock_ni; +pub mod bedrock_xb; +pub mod bedrock_lb; +pub mod bedrock_md; +pub mod bedrock_mem; +pub mod bedrock_pi; pub mod dp83843; pub mod ics1890; -pub mod timekeeper; +pub mod isp12160; +pub mod l1; +pub mod ohci; pub mod prom_image; +pub mod timekeeper; +pub mod xbridge; diff --git a/src/dev/odyssey/board.rs b/src/dev/odyssey/board.rs index 8a3cb1dbc..c40746087 100644 --- a/src/dev/odyssey/board.rs +++ b/src/dev/odyssey/board.rs @@ -1,5 +1,5 @@ //! The Odyssey (VPro) board's constants and state. The interrupt packet path -//! is the `irq` field of the host interface (`HostInterface`, interface.rs). +//! is the `packets` field of the host interface (`HostInterface`, interface.rs). //! //! Ported from include/hw/misc/sgi-odyssey.h in jamesbraid/qemu-sgi-origami. diff --git a/src/dev/odyssey/debug.rs b/src/dev/odyssey/debug.rs index ce8b6f677..bb40c2d07 100644 --- a/src/dev/odyssey/debug.rs +++ b/src/dev/odyssey/debug.rs @@ -463,7 +463,7 @@ impl Odyssey { f.route_dest_upper, f.route_dest_lower, f.route_vectors, (f.route_vectors >> 24) & 0xff, (f.route_vectors >> 16) & 0xff, (f.route_vectors >> 8) & 0xff, f.route_vectors & 0xff, f.routed()); let _ = writeln!(out, " swap arm {:#x} pending {:#x} control {:#x} companion {:#x} set {:#x} clear {:#x} displayed {:#010x}{}", f.swap_arm, f.swap_pending, f.swap_control, f.swap_control_companion, f.swap_set, f.swap_clear, f.displayed_buffer, if f.displayed_written { " (written, repaint pending)" } else { "" }); - let _ = writeln!(out, " sw token {:#x} flow-control target {:#x} heart widget {}", f.sw_token, f.flctl_target, f.heart_widget); + let _ = writeln!(out, " sw token {:#x} flow-control target {:#x}", f.sw_token, f.flctl_target); } let g = self.quiesced()?; let wg = &g.widget; diff --git a/src/dev/odyssey/fifo.rs b/src/dev/odyssey/fifo.rs index f582e8cf0..cd7183cdb 100644 --- a/src/dev/odyssey/fifo.rs +++ b/src/dev/odyssey/fifo.rs @@ -174,6 +174,12 @@ pub(super) struct CommandFifo { /// counted after its last look at the hurry request. #[cfg(test)] pub test_parked: AtomicU32, + /// Tests: pushes the full ring has refused. + #[cfg(test)] + pub test_refused: AtomicU32, + /// Tests: ends a long slice's hold (`Board::test_slice_sleep_ms`) early. + #[cfg(test)] + pub test_release: AtomicBool, /// Tests: the next push calls this, once, as soon as its /// store is in the ring: a consumer that takes the store at that /// moment, before the push has returned. @@ -205,6 +211,10 @@ impl Default for CommandFifo { #[cfg(test)] test_parked: AtomicU32::new(0), #[cfg(test)] + test_refused: AtomicU32::new(0), + #[cfg(test)] + test_release: AtomicBool::new(false), + #[cfg(test)] test_after_ring_push: Mutex::new(None), } } @@ -334,6 +344,8 @@ impl CommandFifo { pub fn push(&self, front: &Mutex, addr: u64, value: u64, size: u32) -> bool { let words = store_words(size); if !self.has_room() { + #[cfg(test)] + self.test_refused.fetch_add(1, SeqCst); return false; } let q = QueuedStore { value, addr: addr as u32, size: size as u8 }; diff --git a/src/dev/odyssey/interface.rs b/src/dev/odyssey/interface.rs index 49727c3a8..26fe1557a 100644 --- a/src/dev/odyssey/interface.rs +++ b/src/dev/odyssey/interface.rs @@ -29,14 +29,13 @@ use super::display_fifo::display_fifo_access; use super::fifo::CommandFifo; use super::mmio::{access_waits, command_store, fifo_port}; use super::registers::reg; +use super::PacketSender; use parking_lot::Mutex; use std::sync::Arc; pub(super) struct HostInterface { - /// Where interrupt packets go (the iris glue's HEART). - pub irq: Option>, - /// The widget number of HEART, the packets' destination. - pub heart_widget: u32, + /// Where interrupt packets go: the machine's fabric. + pub packets: Option, /// ODSY_IDENTITY, composed from the revision at reset. pub identity: u32, @@ -88,8 +87,7 @@ pub(super) struct HostInterface { impl Default for HostInterface { fn default() -> Self { HostInterface { - irq: None, - heart_widget: 8, + packets: None, identity: 0, widget_control: 0, xt_req_timeout: 0, @@ -498,18 +496,28 @@ pub(super) fn sgi_xio_compose(who: &str, widget: u32, address: u64, packet: &mut /// HEART's interrupt-packet register in its widget space. const HEART_PACKET: u64 = 0x80; +/// The Octane's packet sender: HEART takes a packet at 0x80 of widget 8 and +/// drives the line the vector names. +pub(super) fn heart_sender(irq: Arc) -> PacketSender { + Arc::new(move |widget, address, vector, level| { + if widget != crate::ip30::WIDGET_HEART || address != HEART_PACKET { + return false; + } + irq.set_line(vector as u32, level); + true + }) +} + impl HostInterface { - /// The interrupt-modify write to the interrupt target. HEART takes it at - /// 0x80 of widget 8. + /// The interrupt-modify write to the interrupt target: the packet goes to + /// the machine's sender, which routes it by widget and address. pub fn send_interrupt(&mut self, packet: u64, number: u8, set: bool) { let widget = (packet >> XIO_PACKET_ADDRESS_BITS) as u32; let address = packet & ((1u64 << XIO_PACKET_ADDRESS_BITS) - 1); - if widget != self.heart_widget || address != HEART_PACKET { - crate::dlog!(LogModule::Odyssey, "odyssey: interrupt packet to widget {widget} address {address:#x} has no receiver"); - return; - } - if let Some(irq) = &self.irq { - irq.set_line(number as u32, set); + if let Some(send) = &self.packets { + if !send(widget as u8, address, number, set) { + crate::dlog!(LogModule::Odyssey, "odyssey: interrupt packet to widget {widget} address {address:#x} has no receiver"); + } } } } diff --git a/src/dev/odyssey/mod.rs b/src/dev/odyssey/mod.rs index e5cad6423..6d036f9c6 100644 --- a/src/dev/odyssey/mod.rs +++ b/src/dev/odyssey/mod.rs @@ -127,19 +127,34 @@ use scanout::monotonic_ns; /// The VPro board on widget `slot`: a `BusDevice` for the widget's /// windows. pub struct Odyssey { - small: u64, - large: u64, + aperture: Aperture, state: BoardLocks, display: Arc, running: Arc, thread: Mutex>>, } +/// Carries a board interrupt packet into a machine's crosstalk fabric: the +/// destination widget, the address within it, the vector, and whether the +/// source rose or fell. False when nothing at the destination takes it. +pub type PacketSender = Arc bool + Send + Sync>; + +/// How the processor's addresses reach the board, chosen when it is built. +#[derive(Clone, Copy)] +enum Aperture { + /// The Octane's: the widget's small and large IP30 windows. + Ip30Windows { small: u64, large: u64 }, + /// Every address is already an aperture offset, as a machine whose I/O + /// interface hands widgets offsets (Bedrock) presents it. + Offsets, +} + impl Odyssey { - /// A board of `product` (v12, v10, v8, v6) at `widget`: its interrupt - /// packets reach `irq` (HEART), its DMA reaches `mem`. `pool` is the - /// CPU raster unit's workers and tile size, `raster` which unit draws - /// and, for the GPU, the wgpu backend (`[graphics]`). + /// A board of `product` (v12, v10, v8, v6) at `widget` on the Octane: + /// the processor reaches it through the IP30 widget windows, its + /// interrupt packets reach `irq` (HEART), its DMA reaches `mem`. `pool` + /// is the CPU raster unit's workers and tile size, `raster` which unit + /// draws and, for the GPU, the wgpu backend (`[graphics]`). #[allow(clippy::too_many_arguments, reason = "the board's wiring")] pub fn new( widget: u8, @@ -149,6 +164,37 @@ impl Odyssey { status: Arc, pool: (usize, u32), raster: (crate::config::RasterUnit, &str), + ) -> Result, String> { + let aperture = Aperture::Ip30Windows { small: widget_small_base(widget), large: widget_large_base(widget) }; + Self::build(widget, product, interface::heart_sender(irq), aperture, mem, status, pool, raster) + } + + /// A board of `product` at `widget` on a machine that hands the board + /// widget-relative addresses, as Bedrock's I/O interface does, and + /// carries its interrupt packets through `packets`. + #[allow(clippy::too_many_arguments, reason = "the board's wiring")] + pub fn new_widget_relative( + widget: u8, + product: &str, + packets: PacketSender, + mem: Arc, + status: Arc, + pool: (usize, u32), + raster: (crate::config::RasterUnit, &str), + ) -> Result, String> { + Self::build(widget, product, packets, Aperture::Offsets, mem, status, pool, raster) + } + + #[allow(clippy::too_many_arguments, reason = "the board's wiring")] + fn build( + widget: u8, + product: &str, + packets: PacketSender, + aperture: Aperture, + mem: Arc, + status: Arc, + pool: (usize, u32), + raster: (crate::config::RasterUnit, &str), ) -> Result, String> { let mut s = Board::new(product, widget as u32)?; // The fragment pipeline's workers and tile size (`[graphics]`). @@ -168,15 +214,14 @@ impl Odyssey { }); let mem: Arc = Arc::new(record::RecordingMem { inner: mem, rec: rec.clone() }); s.guest = Some(mem); - s.front.lock().irq = Some(irq); + s.front.lock().packets = Some(packets); if let Ok(path) = std::env::var("IRIS_ODSY_TRACE") { s.trace.start(&path, trace::TRACE_ALL).map_err(|e| format!("IRIS_ODSY_TRACE {path}: {e}"))?; } let mut state = BoardLocks::new(s); state.rec = rec; Ok(Arc::new(Self { - small: widget_small_base(widget), - large: widget_large_base(widget), + aperture, state, display: Arc::new(Display::new(status)), running: Arc::new(AtomicBool::new(false)), @@ -192,7 +237,10 @@ impl Odyssey { /// The aperture offset of `addr`, in either window. fn offset(&self, addr: u64) -> Option { - let off = if addr >= self.large { addr - self.large } else { addr.wrapping_sub(self.small) }; + let off = match self.aperture { + Aperture::Ip30Windows { small, large } => if addr >= large { addr - large } else { addr.wrapping_sub(small) }, + Aperture::Offsets => addr, + }; (off < ODSY_APERTURE).then_some(off) } diff --git a/src/dev/odyssey/tests/fifo.rs b/src/dev/odyssey/tests/fifo.rs index 8e90e12eb..425a023a0 100644 --- a/src/dev/odyssey/tests/fifo.rs +++ b/src/dev/odyssey/tests/fifo.rs @@ -951,13 +951,14 @@ fn a_refused_command_push_retries_behind_the_board_thread() { let (h, _rec) = threaded_board(); let words = preamble_twice(); - // The ring must fill and the retry be seen waiting while this one slice - // holds the board, however long the test's threads wait for a core. - start_long_slice(&h, &words, 4000); + // The slice holds the board's thread until the test releases it, so the + // ring stays full however long the host takes to get here. + start_long_slice(&h, &words, 600_000); while h.fifo.stores() < RING_STORES { assert!(h.bus_write(consts::cmd::FIFO, 0, 4)); } let stores = h.fifo.stores(); + let refused = h.fifo.test_refused.load(SeqCst); let done = Arc::new(std::sync::atomic::AtomicBool::new(false)); let store = { let (h, done) = (h.clone(), done.clone()); @@ -966,9 +967,15 @@ fn a_refused_command_push_retries_behind_the_board_thread() { done.store(true, SeqCst); }) }; - std::thread::sleep(std::time::Duration::from_millis(150)); + // The store is refused and retries: wait until it has been. + let deadline = std::time::Instant::now() + std::time::Duration::from_secs(20); + while h.fifo.test_refused.load(SeqCst) == refused { + assert!(std::time::Instant::now() < deadline, "the store was never refused"); + std::thread::yield_now(); + } assert!(!done.load(SeqCst), "the store was dropped or ran around the board's thread"); assert_eq!(h.fifo.stores(), stores); + h.fifo.test_release.store(true, SeqCst); store.join().unwrap(); assert!(h.fifo.level() <= cfifo::WORDS); h.fifo.test_words.store(0, SeqCst); diff --git a/src/dev/odyssey/tests/interrupts.rs b/src/dev/odyssey/tests/interrupts.rs index 6bbb67ca9..8f89604cf 100644 --- a/src/dev/odyssey/tests/interrupts.rs +++ b/src/dev/odyssey/tests/interrupts.rs @@ -437,3 +437,26 @@ fn a_held_flag_is_sent_with_the_source_enables_clear() { reg_write(&mut s, route::DEST_UPPER, HEART_UPPER); assert_eq!(rec.events(FLAG_VECTOR), [true]); } + +/// A board built for a machine whose fabric carries packets to any widget +/// hands them to that machine's sender, to the widget the routing names, +/// and never to HEART's line: the Fuel's widget 13 reaches Bedrock this way. +#[test] +fn a_packet_sender_takes_the_interrupt_to_the_routed_widget() { + let (mut s, rec) = board("v12"); + let seen = Arc::new(Mutex::new(Vec::new())); + let log = seen.clone(); + s.front.lock().packets = Some(Arc::new(move |widget, address, vector, level| { + log.lock().push((widget, address, vector, level)); + true + })); + s.mmio_write(ROUTE_DEST_UPPER, 1 << 25, 4); + s.mmio_write(ROUTE_DEST_LOWER, 0x1230, 4); + s.mmio_write(ROUTE_VECTORS, TEST_VECTOR as u64, 4); + s.mmio_write(INTR_ENABLE_SET, INTR_FIFO_LOW_WATER, 4); + assert_eq!(*seen.lock(), vec![(1, 0x1230, TEST_VECTOR as u8, true)]); + assert!(rec.silent(), "the packet did not go to HEART"); + + s.mmio_write(INTR_CLEAR, INTR_FIFO_LOW_WATER, 4); + assert_eq!(seen.lock().last(), Some(&(1, 0x1230, TEST_VECTOR as u8, false))); +} diff --git a/src/dev/odyssey/tests/mod.rs b/src/dev/odyssey/tests/mod.rs index ff2f655b9..96ea31208 100644 --- a/src/dev/odyssey/tests/mod.rs +++ b/src/dev/odyssey/tests/mod.rs @@ -88,7 +88,7 @@ pub(super) fn board(product: &str) -> (Box, Arc) { let rec = Arc::new(Recorder::default()); let mut s = Board::new(product, VPRO_WIDGET).expect("board"); s.test_screen(); - s.front.lock().irq = Some(rec.clone()); + s.front.lock().packets = Some(super::interface::heart_sender(rec.clone())); (s, rec) } diff --git a/src/dev/odyssey/tests/monitor.rs b/src/dev/odyssey/tests/monitor.rs index 01c33271c..9fdbae56f 100644 --- a/src/dev/odyssey/tests/monitor.rs +++ b/src/dev/odyssey/tests/monitor.rs @@ -27,6 +27,12 @@ fn board() -> Arc { v } +/// The base of the board's large IP30 window, where a processor's stores +/// reach its aperture. +fn large() -> u64 { + crate::ip30::widget_large_base(WIDGET) +} + /// The monitor's writer: what a command wrote, read back after it. #[derive(Clone, Default)] struct Sink(Arc>>); @@ -289,9 +295,9 @@ fn rec_starts_marks_and_stops_a_recording_from_the_monitor() { // Two command stores and a front-register load through the bus, as a // processor makes them; then a pixel lands and a mark takes it. - assert_eq!(v.write32(v.large + cmd::FIFO, cmd::REGISTER_TAG | reg::STROBE), crate::traits::BUS_OK); - assert_eq!(v.write32(v.large + cmd::FIFO, cmd::REGISTER_TAG | reg::STROBE), crate::traits::BUS_OK); - let status0 = v.read32(v.large + ODSY_STATUS0); + assert_eq!(v.write32(large() + cmd::FIFO, cmd::REGISTER_TAG | reg::STROBE), crate::traits::BUS_OK); + assert_eq!(v.write32(large() + cmd::FIFO, cmd::REGISTER_TAG | reg::STROBE), crate::traits::BUS_OK); + let status0 = v.read32(large() + ODSY_STATUS0); assert!(status0.is_ok()); pixel_write(v.state.test_board(), 3, 4, 0x1234_5600); let h1 = memory_hash_bytes(v.state.test_board()); @@ -353,12 +359,12 @@ fn trace_needs_gputrace_and_then_writes_the_classes_asked_for() { // A GL begin and three integer vertices through the kernel FIFO port: the // words are FIFO lines, the third vertex completes a triangle (a DRAW // line); three identical status polls are one CPU line. - let fifo = v.large + cmd::FIFO; + let fifo = large() + cmd::FIFO; for word in [cmd::REGISTER_TAG | reg::PATTERN_MODE, 1 << 13, cmd::REGISTER_TAG | reg::GL_BEGIN, gl_cmd::TRIANGLES, gl_cmd::VERTEX_2I, 10, 20, gl_cmd::VERTEX_2I, 30, 20, gl_cmd::VERTEX_2I, 10, 40] { assert_eq!(v.write32(fifo, word), crate::traits::BUS_OK); } for _ in 0..3 { - assert!(v.read32(v.large + ODSY_STATUS0).is_ok()); + assert!(v.read32(large() + ODSY_STATUS0).is_ok()); } assert_eq!(ok(&v, "trace flush").trim(), "trace flushed"); let text = std::fs::read_to_string(&path).unwrap(); diff --git a/src/dev/odyssey/worker.rs b/src/dev/odyssey/worker.rs index b0a44abf0..1f60af2c0 100644 --- a/src/dev/odyssey/worker.rs +++ b/src/dev/odyssey/worker.rs @@ -141,7 +141,11 @@ impl Board { // A run that takes as long as a long draw would. let sleep_ms = std::mem::take(&mut self.test_slice_sleep_ms); if sleep_ms != 0 { - std::thread::sleep(std::time::Duration::from_millis(sleep_ms as u64)); + // Until the time is up or a test releases the hold. + let end = std::time::Instant::now() + std::time::Duration::from_millis(sleep_ms as u64); + while std::time::Instant::now() < end && !self.fifo.test_release.load(SeqCst) { + std::thread::sleep(std::time::Duration::from_millis(1)); + } } } diff --git a/src/dev/ohci.rs b/src/dev/ohci.rs new file mode 100644 index 000000000..edbbb6c62 --- /dev/null +++ b/src/dev/ohci.rs @@ -0,0 +1,584 @@ +// Derived from MAME's xbox_usb.cpp and xbox_usb.h (the OHCI host controller), +// BSD-3-Clause, copyright Samuele Zannoli. + +//! The Lucent USS-312 USB controller: two root ports, OHCI registers and +//! descriptor processing through the PCI bridge's DMA interface. +//! +//! Functions plug into a root port with `attach`: the Fuel fits a keyboard and +//! a mouse (`usb_hid`), and the tests drive the descriptor and packet +//! scheduling engine through a test-only function. Isochronous +//! traffic, suspend/resume, external hubs, USB storage and legacy keyboard/SMI +//! routing are unsupported. Scheduling uses synthetic work units rather than +//! wire time. Register behavior follows OpenHCI 1.0a and the USS-312 revision 5 +//! data sheet. +//! +//! Ported from src/mame/shared/xbox_usb.cpp and xbox_usb.h in MAME, +//! revision 2ee9b1e30299b122b131cab0bd7912abc2b095ba. + +use crate::dev::heart::{host_clock, Clock}; +use crate::irq::IrqLine; +use crate::pci::{Bar, PciConfig, PciConfigPolicy, PciDma, PciFunction, PciFunctionState, PCI_INTA}; +use crate::traits::{Device, Resettable}; +use parking_lot::Mutex; +use std::sync::atomic::{AtomicBool, Ordering}; +use std::sync::{Arc, OnceLock}; + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +#[repr(u32)] +pub enum Pid { + Setup = 0, + Out = 1, + In = 2, +} +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct UsbPacket { + pub endpoint: u8, + pub pid: Pid, + pub toggle: bool, + /// IN requests name capacity; OUT and SETUP carry their bytes. + pub capacity: usize, + pub data: Vec, + pub time_ms: u64, +} +#[derive(Clone, Debug, PartialEq, Eq)] +pub enum UsbOutcome { + Ack(Vec), + Nak, + Stall, + NoResponse, + CrcError, + BitStuffError, +} +pub trait UsbFunction: Send + Sync { + fn reset(&self); + fn address(&self) -> u8; + fn low_speed(&self) -> bool { + false + } + fn transact(&self, packet: UsbPacket) -> UsbOutcome; +} + +mod engine; +mod schedule; + +const EVENTS: u32 = 0x4000_007f; +const WDH: u32 = 2; +const UE: u32 = 16; +const RHSC: u32 = 64; +const PPS: u32 = 0x100; +const CHANGES: u32 = 0x1f0000; +#[derive(Clone)] +struct State { + r: [u32; 23], + ports: [Option>; 2], + low_speed: [bool; 2], + reset_ms: [u8; 2], + done_delay: u8, + now_ms: u64, + active_fi: u32, + bit_credit: u64, +} +impl State { + fn new() -> Self { + let mut r = [0; 23]; + r[0] = 0x110; + r[13] = 0x2edf; + r[17] = 0x628; + r[18] = 0x1000_0902; + r[19] = 0x60000; + Self { + r, + ports: [None, None], + low_speed: [false; 2], + reset_ms: [0; 2], + done_delay: 7, + now_ms: 0, + active_fi: 0x2edf, + bit_credit: 0, + } + } + fn soft_reset(&mut self) { + let control = (self.r[1] & 0x300) | 0xc0; + let keep = [self.r[18], self.r[19], self.r[20], self.r[21], self.r[22]]; + let ports = self.ports.clone(); + let low_speed = self.low_speed; + let resets = self.reset_ms; + let time = self.now_ms; + *self = Self::new(); + self.r[1] = control; + self.r[18..23].copy_from_slice(&keep); + self.ports = ports; + self.low_speed = low_speed; + self.reset_ms = resets; + self.now_ms = time; + } + fn irq(&self) -> bool { + self.r[1] & 0x100 == 0 && self.r[4] & 0x80000000 != 0 && self.r[3] & self.r[4] & EVENTS != 0 + } + fn powered(&self, p: usize) -> bool { + self.r[18] & 0x200 != 0 || self.r[21 + p] & PPS != 0 + } + fn individual(&self, p: usize) -> bool { + self.r[18] & 0x100 != 0 && self.r[19] & (1 << (17 + p)) != 0 + } + fn power(&mut self, p: usize, on: bool) { + if self.r[18] & 0x200 != 0 || on { + self.r[21 + p] |= PPS; + if self.ports[p].is_some() && self.r[21 + p] & 1 == 0 { + self.r[21 + p] |= 1 | 0x10000 | if self.low_speed[p] { 0x200 } else { 0 }; + self.r[3] |= RHSC; + } + } else { + let previous = self.r[21 + p]; + self.r[21 + p] &= !(PPS | 0x17); + if previous & 1 != 0 { + self.r[21 + p] |= 0x10000; + } + if previous & 2 != 0 { + self.r[21 + p] |= 0x20000; + } + if previous & 3 != 0 { + self.r[3] |= RHSC; + } + self.reset_ms[p] = 0; + } + } +} + +pub struct Ohci { + pci: PciFunctionState, + state: Mutex, + pass: Mutex<()>, + running: AtomicBool, + /// Source of the frame count: the machine's own clock, so the guest's + /// idea of a millisecond and the controller's agree. + time: OnceLock, + /// `time` reading at the last frame boundary delivered; `None` while stopped. + clock: Mutex>, +} +impl Ohci { + pub fn new() -> Arc { + let config = PciConfig::new_with_policy( + 0x11c1, + 0x5802, + 0x0c0310, + 0x10, + [ + Bar::Mem(0x1000), + Bar::Unused, + Bar::Unused, + Bar::Unused, + Bar::Unused, + Bar::Unused, + ], + PCI_INTA, + PciConfigPolicy { + status: 0x0200, + status_w1c_mask: 0x8000, + command_mask: 0x0006, + latency_mask: 0, + cache_line_mask: 0, + registers: &[], + }, + ); + config.write(11, 0xffff_ffff); + Arc::new(Self { + pci: PciFunctionState::new(config), + state: Mutex::new(State::new()), + pass: Mutex::new(()), + running: AtomicBool::new(false), + time: OnceLock::new(), + clock: Mutex::new(None), + }) + } + /// Use `clock` for frame timing. Call before `start`. + pub fn set_clock(&self, clock: Clock) { + assert!( + !self.running.load(Ordering::Acquire) && self.time.set(clock).is_ok(), + "connect the OHCI clock once, before start" + ); + } + fn now(&self) -> u64 { + (self.time.get_or_init(host_clock))() + } + /// Deliver every frame boundary the clock has passed. Every register + /// access calls this, so a read sees the frame count the guest's own + /// clock implies, however late the machine's poll or timer ran. + pub fn sync_clock(&self) { + // Held through delivery: a second caller must not find the boundary + // consumed and read the frame count before the first has applied it. + let mut clock = self.clock.lock(); + let Some(at) = clock.as_mut() else { return }; + let now = self.now(); + let elapsed = now.saturating_sub(*at) / 1_000_000; + *at += elapsed * 1_000_000; + self.advance_frames(elapsed); + } + fn drive_irq(&self) { + loop { + let level = self.state.lock().irq(); + if let Some(line) = self.pci.irq() { + line.set(level); + } + // A callback may acknowledge the interrupt or another writer may + // change it while delivery is outside the state lock. + if self.state.lock().irq() == level { + break; + } + } + } + pub fn attach(&self, port: usize, function: Arc) -> Result<(), String> { + if port >= 2 { + return Err("OHCI has two ports".into()); + } + if self.state.lock().ports[port].is_some() { + return Err("OHCI port occupied".into()); + } + let low_speed = function.low_speed(); + function.reset(); + { + let _pass = self.pass.lock(); + let mut s = self.state.lock(); + if s.ports[port].is_some() { + return Err("OHCI port occupied".into()); + } + if s.powered(port) { + s.r[21 + port] |= 1 | 0x10000; + } + s.low_speed[port] = low_speed; + if low_speed { + s.r[21 + port] |= 0x200; + } + s.ports[port] = Some(function); + s.r[3] |= RHSC; + } + self.drive_irq(); + Ok(()) + } + #[cfg(test)] + pub fn detach(&self, port: usize) -> Result<(), String> { + if port >= 2 { + return Err("OHCI has two ports".into()); + } + { + let _pass = self.pass.lock(); + let mut s = self.state.lock(); + let enabled = s.r[21 + port] & 2; + s.ports[port] = None; + s.low_speed[port] = false; + s.reset_ms[port] = 0; + s.r[21 + port] = + (s.r[21 + port] & PPS) | 0x10000 | if enabled != 0 { 0x20000 } else { 0 }; + s.r[3] |= RHSC; + } + self.drive_irq(); + Ok(()) + } + /// Deterministic millisecond ticks; `sync_clock` delivers the same ticks from the clock. + pub fn advance_frames(&self, count: u64) { + for _ in 0..count { + if !self.running.load(Ordering::Acquire) { + break; + } + { + let _pass = self.pass.lock(); + if !self.running.load(Ordering::Acquire) { + break; + } + let mut s = self.state.lock().clone(); + self.frame(&mut s); + *self.state.lock() = s; + } + self.drive_irq(); + } + } + fn frame(&self, s: &mut State) { + s.now_ms = s.now_ms.wrapping_add(1); + for p in 0..2 { + if s.reset_ms[p] != 0 { + s.reset_ms[p] -= 1; + if s.reset_ms[p] == 0 { + s.r[21 + p] = (s.r[21 + p] & !16) | 2 | (1 << 20); + s.r[3] |= RHSC; + } + } + } + if s.r[1] & 0xc0 != 0x80 || s.r[3] & UE != 0 { + return; + } + s.bit_credit += 12000; + while s.bit_credit >= u64::from((s.active_fi & 0x3fff) + 1) { + s.bit_credit -= u64::from((s.active_fi & 0x3fff) + 1); + s.active_fi = s.r[13]; + if self.schedule(s).is_err() { + s.r[3] |= UE; + s.bit_credit = 0; + self.pci.config().set_status_bits(0x8000); + break; + } + } + } +} +impl PciFunction for Ohci { + fn state(&self) -> &PciFunctionState { + &self.pci + } + fn attached(&self) { + self.drive_irq(); + } + fn reset(&self) { + Resettable::power_on(self); + } + fn bar_read(&self, bar: usize, off: u32, width: usize) -> u32 { + if bar != 0 || width != 4 || off & 3 != 0 || off > 0x58 { + crate::dlog!( + crate::devlog::LogModule::Usb, + "ohci unsupported read at register offset {off:#x}, width {width}" + ); + return 0; + } + self.sync_clock(); + let s = self.state.lock(); + let i = (off / 4) as usize; + if i == 5 { + s.r[4] + } else if i == 21 || i == 22 { + s.r[i] | if s.r[18] & 0x200 != 0 { PPS } else { 0 } + } else { + s.r[i] + } + } + fn bar_write(&self, bar: usize, off: u32, val: u32, width: usize) { + if bar != 0 || width != 4 || off & 3 != 0 || off > 0x58 { + crate::dlog!( + crate::devlog::LogModule::Usb, + "ohci unsupported write at register offset {off:#x}, width {width}" + ); + return; + } + self.sync_clock(); + // Taken before `pass`, as `sync_clock` orders them. Only a control + // write can restart the frame phase. + let mut anchor = (off == 4).then(|| self.clock.lock()); + let mut reset = None; + { + let _pass = self.pass.lock(); + let mut s = self.state.lock(); + match off / 4 { + 1 => { + let old = s.r[1]; + s.r[1] = val & 0x7ff; + if val & 0xc0 == 0x80 && old & 0xc0 != 0x80 { + // OpenHCI 1.0a 6.2.1, 7.3.2, 7.3.3: entering + // UsbOperational loads HcFmRemaining from the frame + // interval and increments HcFmNumber, with no SOF at + // the load; the first follows one frame later. The + // free-running clock's phase therefore restarts here. + s.bit_credit = 0; + s.active_fi = s.r[13]; + s.r[14] = (s.r[13] & 0x8000_0000) | (s.r[13] & 0x3fff); + let old_frame = s.r[15] as u16; + let frame = old_frame.wrapping_add(1); + s.r[15] = u32::from(frame); + if (old_frame ^ frame) & 0x8000 != 0 { + s.r[3] |= 32; + } + if let Some(Some(at)) = anchor.as_deref_mut() { + *at = self.now(); + } + } + if val & 0xc0 == 0 && old & 0xc0 != 0 { + s.r[18] = 0x10000902; + for p in 0..2 { + s.power(p, false); + } + } + if val & 0xc0 == 0x40 || val & 0xc0 == 0xc0 { + crate::dlog!( + crate::devlog::LogModule::Usb, + "ohci suspend/resume timing unsupported at control register offset 0x04" + ); + } + } + 2 => { + if val & 1 != 0 { + s.soft_reset(); + } else { + s.r[2] |= val & 6; + if val & 8 != 0 { + s.r[3] |= 1 << 30; + } + } + } + 3 => s.r[3] &= !(val & EVENTS), + 4 => s.r[4] |= val & (EVENTS | 0x80000000), + 5 => s.r[4] &= !(val & (EVENTS | 0x80000000)), + 6 => s.r[6] = val & !255, + 7..=11 => s.r[(off / 4) as usize] = val & !15, + 13 => s.r[13] = val & 0xffff3fff, + 16 => s.r[16] = val & 0x3fff, + 17 => s.r[17] = val & 0xfff, + 18 => { + s.r[18] = (s.r[18] & !0xff001b00) | (val & 0xff001b00); + if val & 0x200 != 0 { + for p in 0..2 { + s.power(p, true); + } + } + } + 19 => s.r[19] = val & 0x60006, + 20 => { + if val & (1 << 15) != 0 { + s.r[20] |= 1 << 15; + } + if val & (1 << 31) != 0 { + s.r[20] &= !(1 << 15); + } + if val & (1 << 17) != 0 { + s.r[20] &= !(1 << 17); + } + if s.r[18] & 0x200 == 0 { + for p in 0..2 { + if !s.individual(p) { + if val & 1 != 0 { + s.power(p, false); + } + if val & (1 << 16) != 0 { + s.power(p, true); + } + } + } + } + } + 21 | 22 => { + let p = (off / 4 - 21) as usize; + let i = 21 + p; + s.r[i] &= !(val & CHANGES); + if val & 1 != 0 { + s.r[i] &= !2; + } + if val & 2 != 0 { + if s.r[i] & 1 != 0 && s.powered(p) { + s.r[i] |= 2; + } else { + s.r[i] |= 1 << 16; + } + } + if val & 12 != 0 { + crate::dlog!( + crate::devlog::LogModule::Usb, + "ohci port suspend/resume unsupported at port register offset {:#x}", + 0x54 + p * 4 + ); + } + if val & 16 != 0 { + if s.r[i] & 1 != 0 && s.powered(p) { + s.r[i] |= 16; + s.r[i] &= !2; + s.reset_ms[p] = 10; + reset = s.ports[p].clone(); + } else { + s.r[i] |= 1 << 16; + } + } + if s.individual(p) && s.r[18] & 0x200 == 0 { + if val & PPS != 0 { + s.power(p, true); + } + if val & 0x200 != 0 { + s.power(p, false); + } + } + if s.r[i] & CHANGES != 0 { + s.r[3] |= RHSC; + } + } + _ => { + crate::dlog!( + crate::devlog::LogModule::Usb, + "ohci write to read-only register offset {off:#x}" + ); + } + } + } + drop(anchor); + if let Some(f) = reset { + f.reset(); + } + self.drive_irq(); + } +} +impl Resettable for Ohci { + fn power_on(&self) { + let ports = { + let _pass = self.pass.lock(); + let mut s = self.state.lock(); + let ports = s.ports.clone(); + let low_speed = s.low_speed; + *s = State::new(); + s.ports = ports.clone(); + s.low_speed = low_speed; + ports + }; + for f in ports.into_iter().flatten() { + f.reset(); + } + self.drive_irq(); + } +} +impl Device for Ohci { + fn start(&self) { + if self.running.swap(true, Ordering::AcqRel) { + return; + } + let now = self.now(); + *self.clock.lock() = Some(now); + } + fn stop(&self) { + self.sync_clock(); + self.running.store(false, Ordering::Release); + *self.clock.lock() = None; + drop(self.pass.lock()); + } + fn is_running(&self) -> bool { + self.running.load(Ordering::Acquire) + } + fn step(&self, _: u64) {} + fn get_clock(&self) -> u64 { + 0 + } +} + +#[cfg(test)] +mod register_tests { + use super::*; + use crate::pci::PciFunction; + use crate::traits::Resettable; + + #[test] + fn lucent_identity_and_reset_contract() { + let hc = Ohci::new(); + assert_eq!(hc.state().config().read(0), 0x5802_11c1); + assert_eq!(hc.state().config().read(2), 0x0c03_1010); + assert_eq!(hc.state().config().read(11), 0xffff_ffff); + assert_eq!(hc.state().config().read(1), 0x0200_0000); + assert_eq!(hc.bar_read(0, 0, 4), 0x110); + assert_eq!(hc.bar_read(0, 0x10, 4), 0); + assert_eq!(hc.bar_read(0, 0x34, 4), 0x2edf); + assert_eq!(hc.bar_read(0, 0x48, 4), 0x1000_0902); + assert_eq!(hc.bar_read(0, 0x4c, 4), 0x0006_0000); + for p in [0x54, 0x58] { + assert_eq!(hc.bar_read(0, p, 4), 0); + } + hc.bar_write(0, 0x10, 0xffff_ffff, 4); + hc.bar_write(0, 0x08, 1, 4); + assert_eq!(hc.bar_read(0, 0x10, 4), 0); + assert_eq!(hc.bar_read(0, 0x04, 4) & 0xc0, 0xc0); + hc.power_on(); + assert_eq!(hc.bar_read(0, 0x04, 4), 0); + } +} + +#[cfg(test)] +#[path = "ohci/tests.rs"] +mod tests; diff --git a/src/dev/ohci/engine.rs b/src/dev/ohci/engine.rs new file mode 100644 index 000000000..7b76cc646 --- /dev/null +++ b/src/dev/ohci/engine.rs @@ -0,0 +1,223 @@ +//! General TD packet processing; guest DMA byte slices are the endianness edge. +use super::*; + +fn read_words(dma: &dyn PciDma, at: u32) -> Result<[u32; 4], ()> { + let mut bytes = [0; 16]; + if !dma.dma_read(at as u64, &mut bytes).is_ok() { + return Err(()); + } + Ok(std::array::from_fn(|i| { + u32::from_le_bytes(bytes[i * 4..i * 4 + 4].try_into().unwrap()) + })) +} +fn write_words(dma: &dyn PciDma, at: u32, words: &[u32]) -> Result<(), ()> { + let bytes: Vec = words.iter().flat_map(|x| x.to_le_bytes()).collect(); + if dma.dma_write(at as u64, &bytes).is_ok() { + Ok(()) + } else { + Err(()) + } +} +fn remaining(cbp: u32, be: u32) -> Result { + if cbp == 0 { + return Ok(0); + } + if cbp & !0xfff == be & !0xfff { + be.checked_sub(cbp).map(|n| n as usize + 1).ok_or(()) + } else { + Ok(0x1000 - (cbp as usize & 0xfff) + (be as usize & 0xfff) + 1) + } +} +fn advance(cbp: u32, be: u32, n: usize) -> u32 { + let first = 0x1000 - (cbp as usize & 0xfff); + if n < first { + cbp.wrapping_add(n as u32) + } else { + (be & !0xfff).wrapping_add((n - first) as u32) + } +} +fn payload_read(dma: &dyn PciDma, cbp: u32, be: u32, len: usize) -> Result, ()> { + let mut bytes = vec![0; len]; + let first = len.min(0x1000 - (cbp as usize & 0xfff)); + if first != 0 && !dma.dma_read(cbp as u64, &mut bytes[..first]).is_ok() { + return Err(()); + } + if len > first + && !dma + .dma_read((be & !0xfff) as u64, &mut bytes[first..]) + .is_ok() + { + return Err(()); + } + Ok(bytes) +} +fn payload_write(dma: &dyn PciDma, cbp: u32, be: u32, bytes: &[u8]) -> Result<(), ()> { + let first = bytes.len().min(0x1000 - (cbp as usize & 0xfff)); + if first != 0 && !dma.dma_write(cbp as u64, &bytes[..first]).is_ok() { + return Err(()); + } + if bytes.len() > first && !dma.dma_write((be & !0xfff) as u64, &bytes[first..]).is_ok() { + return Err(()); + } + Ok(()) +} +impl Ohci { + /// Returns the next ED and whether this ED still holds queued general work. + pub(super) fn endpoint( + &self, + s: &mut State, + dma: &dyn PciDma, + at: u32, + ) -> Result<(u32, bool, bool), ()> { + let mut ed = read_words(dma, at)?; + let next = ed[3] & !15; + if ed[0] & (1 << 15) != 0 { + crate::dlog!(crate::devlog::LogModule::Usb, "ohci isochronous ED unsupported at guest address {at:#x}"); + return Ok((next, false, false)); + } + if ed[0] & (1 << 14) != 0 || ed[2] & 1 != 0 || ed[2] & !15 == ed[1] & !15 { + return Ok((next, false, false)); + } + let td_at = ed[2] & !15; + let mut td = read_words(dma, td_at)?; + let mps = ((ed[0] >> 16) & 0x7ff) as usize; + if mps == 0 || mps > 1023 { + crate::dlog!(crate::devlog::LogModule::Usb, "ohci invalid maximum packet size {mps} at ED address {at:#x}"); + return Ok((next, false, false)); + } + let Ok(n) = remaining(td[1], td[3]) else { + crate::dlog!(crate::devlog::LogModule::Usb, "ohci malformed TD buffer range at ED address {at:#x}"); + return Ok((next, false, false)); + }; + let direction = match (ed[0] >> 11) & 3 { + 1 => 1, + 2 => 2, + _ => (td[0] >> 19) & 3, + }; + let pid = match direction { + 0 => Pid::Setup, + 1 => Pid::Out, + 2 => Pid::In, + _ => { + crate::dlog!(crate::devlog::LogModule::Usb, "ohci reserved PID at ED address {at:#x}"); + return Ok((next, false, false)); + } + }; + let length = n.min(mps); + let toggle = if td[0] & (1 << 25) != 0 { + td[0] & (1 << 24) != 0 + } else { + ed[2] & 2 != 0 + }; + let outgoing = if pid != Pid::In { + payload_read(dma, td[1], td[3], length)? + } else { + vec![] + }; + let address = (ed[0] & 127) as u8; + let mut functions = (0..2) + .filter(|&p| s.powered(p) && s.r[21 + p] & 0x16 == 2) + .filter_map(|p| s.ports[p].clone()) + .filter(|f| f.address() == address); + // Multiple matching functions are an unsupported topology. Do not choose + // a recipient by port order or claim to model electrical collisions. + let first = functions.next(); + let function = if functions.next().is_none() { + first + } else { + None + }; + let reply = if let Some(f) = function { + f.transact(UsbPacket { + endpoint: ((ed[0] >> 7) & 15) as u8, + pid, + toggle, + capacity: if pid == Pid::In { mps } else { length }, + data: outgoing, + time_ms: s.now_ms, + }) + } else { + UsbOutcome::NoResponse + }; + let mut retire = false; + let mut cc = 0; + let mut halt = false; + match reply { + UsbOutcome::Nak => return Ok((next, true, true)), + UsbOutcome::Stall => { + cc = 4; + halt = true; + retire = true; + } + UsbOutcome::NoResponse | UsbOutcome::CrcError | UsbOutcome::BitStuffError => { + cc = match reply { + UsbOutcome::CrcError => 1, + UsbOutcome::BitStuffError => 2, + _ => 5, + }; + let ec = ((td[0] >> 26) & 3) + 1; + td[0] = (td[0] & !(3 << 26)) | ((ec.min(3)) << 26); + if ec >= 3 { + retire = true; + halt = true; + } + } + UsbOutcome::Ack(bytes) => { + let actual = if pid == Pid::In { bytes.len() } else { length }; + if pid == Pid::In { + payload_write(dma, td[1], td[3], &bytes[..actual.min(n).min(mps)])?; + } + td[0] = (td[0] & !((3 << 24) | (3 << 26))) | ((2 | u32::from(!toggle)) << 24); + if actual > length { + cc = 8; + halt = true; + retire = true; + } else { + td[1] = if actual == n { + 0 + } else { + advance(td[1], td[3], actual) + }; + if actual < length { + retire = true; + if td[0] & (1 << 18) == 0 { + cc = 9; + halt = true; + } + } else if actual == n { + retire = true; + } + } + } + } + td[0] = (td[0] & 0x0fff_ffff) | (cc << 28); + if retire { + let next_td = td[2] & !15; + td[2] = s.r[12]; + // Failure stops immediately. Earlier successful writes and USB side + // effects remain visible; descriptor writeback is not a transaction. + ed[2] = next_td | ((td[0] >> 23) & 2) | u32::from(halt); + write_words(dma, at + 8, &[ed[2]])?; + write_words(dma, td_at, &td)?; + s.r[12] = td_at; + let di = ((td[0] >> 21) & 7) as u8; + s.done_delay = s.done_delay.min(if cc != 0 { 0 } else { di }); + Ok((next, next_td != ed[1] & !15 && !halt, true)) + } else { + write_words(dma, td_at, &td)?; + Ok((next, true, true)) + } + } + pub(super) fn publish(&self, s: &mut State, dma: &dyn PciDma) -> Result<(), ()> { + if s.done_delay == 0 && s.r[3] & WDH == 0 && s.r[12] != 0 { + let other = u32::from(s.r[3] & s.r[4] & EVENTS & !WDH != 0); + write_words(dma, s.r[6] + 0x84, &[s.r[12] | other])?; + s.r[12] = 0; + s.done_delay = 7; + s.r[3] |= WDH; + } else if s.done_delay > 0 && s.done_delay < 7 { + s.done_delay -= 1; + } + Ok(()) + } +} diff --git a/src/dev/ohci/schedule.rs b/src/dev/ohci/schedule.rs new file mode 100644 index 000000000..5042b9da1 --- /dev/null +++ b/src/dev/ohci/schedule.rs @@ -0,0 +1,98 @@ +//! Functional scheduling approximation: 256 synthetic units per ED visit. +//! Costs deliberately do not represent wire time. See the module's limitations. +use super::*; +const VISIT: u32 = 256; +impl Ohci { + fn list_ready(s: &mut State, head: usize, current: usize, filled: u32) -> bool { + if s.r[current] == 0 && s.r[2] & filled != 0 { + s.r[current] = s.r[head]; + s.r[2] &= !filled; + } + s.r[current] != 0 + } + fn nonperiodic( + &self, + s: &mut State, + dma: &dyn PciDma, + budget: &mut u32, + cutoff: u32, + credit: &mut u32, + ) -> Result<(), ()> { + let mut yielded: Option = None; + while *budget >= VISIT && *budget > cutoff { + let control = s.r[1] & 16 != 0 && Self::list_ready(s, 8, 9, 2); + let bulk = s.r[1] & 32 != 0 && Self::list_ready(s, 10, 11, 4); + if !control && !bulk { + break; + } + let choose_control = control + && (!bulk + || match yielded { + Some(was_control) => !was_control, + None => *credit > 0, + }); + let (current, filled) = if choose_control { (9, 2) } else { (11, 4) }; + let (next, queued, attempt) = self.endpoint(s, dma, s.r[current])?; + *budget -= VISIT; + s.r[current] = next; + if queued { + s.r[2] |= filled; + } + yielded = if attempt { None } else { Some(choose_control) }; + if attempt { + if choose_control { + *credit = credit.saturating_sub(1); + } else { + *credit = (s.r[1] & 3) + 1; + } + } + } + Ok(()) + } + pub(super) fn schedule(&self, s: &mut State) -> Result<(), ()> { + let old = s.r[15] as u16; + let frame = old.wrapping_add(1); + s.r[15] = u32::from(frame); + if (old ^ frame) & 0x8000 != 0 { + s.r[3] |= 32; + } + s.r[3] |= 4; + let mut budget = (s.active_fi & 0x3fff) + 1; + s.r[14] = (s.active_fi & 0x80000000) | budget.min(0x3fff); + if self.pci.config().command() & 4 == 0 { + return Ok(()); + } + let Some(dma) = self.pci.dma().cloned() else { + return Err(()); + }; + let frame_bytes = [frame as u8, (frame >> 8) as u8, 0, 0]; + if !dma.dma_write((s.r[6] + 0x80) as u64, &frame_bytes).is_ok() { + return Err(()); + } + let mut credit = (s.r[1] & 3) + 1; + let cutoff = s.r[16].min(budget); + self.nonperiodic(s, &*dma, &mut budget, cutoff, &mut credit)?; + if s.r[1] & 4 != 0 { + let mut bytes = [0; 4]; + if !dma + .dma_read((s.r[6] + u32::from(frame & 31) * 4) as u64, &mut bytes) + .is_ok() + { + return Err(()); + } + s.r[7] = u32::from_le_bytes(bytes) & !15; + while s.r[7] != 0 && budget >= VISIT { + let (next, _, _) = self.endpoint(s, &*dma, s.r[7])?; + s.r[7] = next; + budget -= VISIT; + } + if s.r[7] != 0 { + s.r[3] |= 1; + s.r[2] = (s.r[2] & !0x30000) | ((s.r[2].wrapping_add(0x10000)) & 0x30000); + } + } + self.nonperiodic(s, &*dma, &mut budget, 0, &mut credit)?; + s.r[14] = (s.active_fi & 0x80000000) | budget.min(0x3fff); + self.publish(s, &*dma) + } +} diff --git a/src/dev/ohci/tests.rs b/src/dev/ohci/tests.rs new file mode 100644 index 000000000..6717287ea --- /dev/null +++ b/src/dev/ohci/tests.rs @@ -0,0 +1,555 @@ +// The register and clock cases follow tests/qtest/usb-lucent-ohci-test.c in +// jamesbraid/qemu-sgi-origami: root-hub-descriptor-a-is-writable and +// frame-number-tracks-clock-while-timer-starved. The descriptor engine cases +// drive a test-only function and have no qtest counterpart. +use super::*; +use crate::pci::{PciDmaStatus, PciFunction}; +use crate::traits::{Device, Resettable}; +use std::sync::atomic::{AtomicUsize, Ordering}; +use std::sync::Weak; + +struct Memory { + bytes: Mutex>, + refuse_read: Mutex>, + refuse_write: Mutex>, + writes: Mutex>, +} +impl Memory { + fn new() -> Arc { + Arc::new(Self { + bytes: Mutex::new(vec![0xa5; 0x10000]), + refuse_read: Mutex::new(None), + refuse_write: Mutex::new(None), + writes: Mutex::new(vec![]), + }) + } + fn words(&self, at: u32, words: &[u32]) { + for (i, w) in words.iter().enumerate() { + self.bytes.lock()[at as usize + i * 4..at as usize + i * 4 + 4] + .copy_from_slice(&w.to_le_bytes()); + } + } + fn word(&self, at: u32) -> u32 { + u32::from_le_bytes( + self.bytes.lock()[at as usize..at as usize + 4] + .try_into() + .unwrap(), + ) + } +} +impl PciDma for Memory { + fn dma_read(&self, addr: u64, dst: &mut [u8]) -> PciDmaStatus { + if self + .refuse_read + .lock() + .is_some_and(|x| (addr..addr + dst.len() as u64).contains(&(x as u64))) + { + return PciDmaStatus::MasterAbort; + } + let b = self.bytes.lock(); + let Some(s) = b.get(addr as usize..addr as usize + dst.len()) else { + return PciDmaStatus::MasterAbort; + }; + dst.copy_from_slice(s); + PciDmaStatus::Ok + } + fn dma_write(&self, addr: u64, src: &[u8]) -> PciDmaStatus { + if self + .refuse_write + .lock() + .is_some_and(|x| (addr..addr + src.len() as u64).contains(&(x as u64))) + { + return PciDmaStatus::MasterAbort; + } + let mut b = self.bytes.lock(); + let Some(s) = b.get_mut(addr as usize..addr as usize + src.len()) else { + return PciDmaStatus::MasterAbort; + }; + s.copy_from_slice(src); + self.writes.lock().push(addr); + PciDmaStatus::Ok + } +} +struct Function { + address: AtomicUsize, + calls: AtomicUsize, + reply: Mutex, + packets: Mutex>, +} +impl Function { + fn new(reply: UsbOutcome) -> Arc { + Arc::new(Self { + address: AtomicUsize::new(0), + calls: AtomicUsize::new(0), + reply: Mutex::new(reply), + packets: Mutex::new(vec![]), + }) + } +} +impl UsbFunction for Function { + fn reset(&self) {} + fn address(&self) -> u8 { + self.address.load(Ordering::SeqCst) as u8 + } + fn transact(&self, packet: UsbPacket) -> UsbOutcome { + self.calls.fetch_add(1, Ordering::SeqCst); + self.packets.lock().push(packet); + self.reply.lock().clone() + } +} +fn start_port_reset(h: &Ohci, port: u32) { + let status = 0x54 + port * 4; + h.bar_write(0, status, 0x100, 4); + h.bar_write(0, status, 0x10, 4); +} + +fn setup( + pid: Pid, + reply: UsbOutcome, + cbp: u32, + be: u32, + flags: u32, +) -> (Arc, Arc, Arc) { + let h = Ohci::new(); + let m = Memory::new(); + let f = Function::new(reply); + h.state().connect_dma(m.clone()); + h.attach(0, f.clone()).unwrap(); + h.start(); + start_port_reset(&h, 0); + h.advance_frames(10); + m.words(0x1000, &[0; 64]); + m.words(0x2000, &[(8 << 16), 0x3020, 0x3000, 0]); + m.words( + 0x3000, + &[0xf000_0000 | flags | ((pid as u32) << 19), cbp, 0x3020, be], + ); + h.state().config().write(1, 6); + h.bar_write(0, 0x18, 0x1000, 4); + h.bar_write(0, 0x20, 0x2000, 4); + h.bar_write(0, 0x08, 2, 4); + h.bar_write(0, 0x04, 0x90, 4); + (h, m, f) +} +/// The guest receives fatal status if the controller cannot access a descriptor. +#[test] +fn dma_refusal_preserves_descriptors_and_reports_a_fatal_error() { + for periodic in [false, true] { + for pid in [Pid::Out, Pid::In] { + let (h, m, f) = setup(pid, UsbOutcome::Ack(vec![5; 8]), 0x4000, 0x4007, 0); + if periodic { + m.words(0x1000, &[0x2000; 32]); + h.bar_write(0, 4, 0x84, 4); + } + if pid == Pid::In { + *m.refuse_write.lock() = Some(0x4000); + } else { + *m.refuse_read.lock() = Some(0x4000); + } + let before = m.bytes.lock()[0x3000..0x3010].to_vec(); + h.advance_frames(1); + let frame = h.bar_read(0, 0x3c, 4); + h.advance_frames(3); + assert_eq!(f.calls.load(Ordering::SeqCst), usize::from(pid == Pid::In)); + assert_eq!(&m.bytes.lock()[0x3000..0x3010], &before); + assert_eq!(m.word(0x2008), 0x3000); + assert_eq!(m.word(0x1084), 0); + assert_eq!(h.bar_read(0, 0x3c, 4), frame); + assert_ne!(h.bar_read(0, 0xc, 4) & UE, 0); + assert_eq!(h.state().config().read(1) >> 16, 0x8200); + h.state().config().write(1, 0x8000_0006); + assert_eq!(h.state().config().read(1) >> 16, 0x0200); + assert_ne!(h.bar_read(0, 0xc, 4) & UE, 0); + } + } +} + +/// Short and overlong USB replies leave bytes outside the packet untouched. +#[test] +fn short_or_overlong_in_packets_preserve_unwritten_bytes() { + for (name, payload, end, flags, bytes, current, condition) in [ + ("short", vec![1, 2, 3], 0x401f, (1 << 18) | (3 << 24), [1, 2, 3, 0xa5, 0xa5], 0x4003, 0), + ("overrun", vec![1; 8], 0x4003, 3 << 24, [1, 1, 1, 1, 0xa5], 0x4000, 8), + ] { + let (h, m, _) = setup(Pid::In, UsbOutcome::Ack(payload), 0x4000, end, flags); + h.advance_frames(1); + assert_eq!(&m.bytes.lock()[0x4000..0x4005], &bytes, "{name}"); + assert_eq!(m.word(0x3004), current, "{name}"); + assert_eq!(m.word(0x3000) >> 28, condition, "{name}"); + assert_eq!((m.word(0x3000) >> 24) & 3, 2, "{name}"); + if condition == 0 { + assert_eq!(m.word(0x2008) & !15, 0x3020); + assert_eq!(m.word(0x1084) & !15, 0x3000); + } + } +} + +/// The guest retries NAKs, stops on STALL, and retains consistent short-transfer state. +#[test] +fn nak_stall_and_short_in_completions_keep_the_endpoint_consistent() { + for (name, reply, flags, condition) in [ + ("NAK", UsbOutcome::Nak, 0, 15), + ("STALL", UsbOutcome::Stall, 3 << 24, 4), + ("no response", UsbOutcome::NoResponse, 0, 5), + ("short without rounding", UsbOutcome::Ack(vec![2]), 0, 9), + ] { + let (h, m, f) = setup(Pid::In, reply, 0x4000, 0x4007, flags); + h.bar_write(0, 0x34, 255, 4); + let before = m.bytes.lock()[0x3000..0x3010].to_vec(); + h.advance_frames(1); + if condition == 5 { + assert_eq!(f.calls.load(Ordering::SeqCst), 1); + assert_eq!((m.word(0x3000) >> 26) & 3, 1); + assert_eq!(m.word(0x2008), 0x3000); + h.advance_frames(2); + } + assert_eq!(m.word(0x3000) >> 28, condition, "{name}"); + if condition == 15 { + assert_eq!(&m.bytes.lock()[0x3000..0x3010], &before); + assert_eq!(m.word(0x2008), 0x3000); + } else { + assert_ne!(m.word(0x2008) & 1, 0, "{name}"); + } + if condition == 4 { + assert_eq!(m.word(0x3004), 0x4000); + assert_eq!((m.word(0x3000) >> 24) & 3, 3); + assert_eq!(m.word(0x2008) & 3, 3); + } + } +} + +/// Port and controller reset paths retain attached USB functions. +#[test] +fn attached_functions_survive_port_and_controller_resets() { + let h = Ohci::new(); + h.start(); + h.attach(1, Function::new(UsbOutcome::Nak)).unwrap(); + assert_eq!(h.bar_read(0, 0x58, 4) & 1, 0); + start_port_reset(&h, 1); + h.advance_frames(9); + assert_ne!(h.bar_read(0, 0x58, 4) & 16, 0); + h.advance_frames(1); + assert_eq!(h.bar_read(0, 0x58, 4) & 16, 0); + assert_ne!(h.bar_read(0, 0x58, 4) & (1 << 20), 0); + h.bar_write(0, 8, 1, 4); + assert_ne!(h.bar_read(0, 0x58, 4) & 1, 0); + h.bar_write(0, 0x58, 0x200, 4); + assert_eq!(h.bar_read(0, 0x58, 4) & 0x117, 0); + h.bar_write(0, 0x58, PPS, 4); + assert_ne!(h.bar_read(0, 0x58, 4) & 1, 0); + h.power_on(); + assert_eq!(h.bar_read(0, 0x58, 4), 0); + h.bar_write(0, 0x58, PPS, 4); + assert_ne!(h.bar_read(0, 0x58, 4) & 1, 0); +} + +/// The guest observes descriptor updates before the done bit and acknowledges them later. +#[test] +fn done_publication_orders_writes_and_waits_for_acknowledgement() { + let (h, m, _) = setup(Pid::In, UsbOutcome::Ack(vec![8; 8]), 0x4000, 0x4007, 0); + h.advance_frames(1); + { + let writes = m.writes.lock(); + assert!(writes.iter().position(|a| *a == 0x2008).unwrap() < writes.iter().position(|a| *a == 0x3000).unwrap()); + } + assert_eq!(m.word(0x1084) & !15, 0x3000); + m.words(0x3020, &[0xf0000000 | (2 << 19), 0x4100, 0x3040, 0x4107]); + m.words(0x2004, &[0x3040]); + h.bar_write(0, 8, 2, 4); + h.advance_frames(1); + assert_eq!(m.word(0x1084) & !15, 0x3000); + assert_eq!(h.bar_read(0, 0x30, 4), 0x3020); + h.bar_write(0, 0xc, 2, 4); + h.advance_frames(1); + assert_eq!(m.word(0x1084) & !15, 0x3020); + for delay in [0, 2, 7] { + let (h, m, _) = setup(Pid::In, UsbOutcome::Ack(vec![5; 8]), 0x4000, 0x4007, delay << 21); + h.advance_frames(1); + if delay != 0 { assert_eq!(m.word(0x1084), 0); } + h.advance_frames(u64::from(delay)); + assert_eq!(m.word(0x1084) & !15, if delay == 7 { 0 } else { 0x3000 }); + } + let (h, m, _) = setup(Pid::In, UsbOutcome::Ack(vec![1; 8]), 0x4000, 0x4007, 0); + *m.refuse_write.lock() = Some(0x1084); + h.advance_frames(1); + assert_ne!(h.bar_read(0, 0xc, 4) & UE, 0); + assert_eq!(h.bar_read(0, 0xc, 4) & WDH, 0); + assert_eq!(m.word(0x1084), 0); +} + +/// OHCI must yield to the other transfer list when a guest marks one list skipped. +#[test] +fn scheduler_shares_service_and_yields_past_skipped_lists() { + for (cbsr, skip_control, skip_bulk) in [(0, false, false), (3, false, false), (3, true, false), (0, false, true)] { + let (h, m, f) = setup(Pid::In, UsbOutcome::Nak, 0x4000, 0x4007, 0); + m.words(0x2000, &[(8 << 16) | (1 << 7) | (u32::from(skip_control) << 14), 0x3020, 0x3000, if skip_control { 0x2000 } else { 0 }]); + m.words(0x2100, &[(8 << 16) | (2 << 7) | (u32::from(skip_bulk) << 14), 0x3120, 0x3100, if skip_bulk { 0x2100 } else { 0 }]); + m.words(0x3100, &[0xf0100000, 0x4100, 0x3120, 0x4107]); + h.bar_write(0, 0x28, 0x2100, 4); + h.bar_write(0, 8, 6, 4); + h.bar_write(0, 4, 0xb0 | cbsr, 4); + h.advance_frames(1); + let packets = f.packets.lock(); + let control = packets.iter().filter(|p| p.endpoint == 1).count(); + let bulk = packets.iter().filter(|p| p.endpoint == 2).count(); + assert_eq!(control > 0, !skip_control); + assert_eq!(bulk > 0, !skip_bulk); + if skip_bulk { + assert!(control > 1, "control resumes after its credit is exhausted"); + } + if !skip_control && !skip_bulk { + assert!(control >= bulk * (cbsr as usize + 1)); + assert!(control <= bulk * (cbsr as usize + 1) + cbsr as usize + 1); + } + assert_eq!(h.bar_read(0, 0xc, 4) & UE, 0); + } +} + +/// The controller routes address zero only while one attached function can claim it. +#[test] +fn address_zero_is_routed_only_to_a_unique_eligible_function() { + for (name, address, peer_status, first_selected, peer_selected) in [ + ("ambiguous", 0, PPS | 1 | 2, false, false), + ("disabled peer", 0, PPS | 1, true, false), + ("assigned first", 5, PPS | 1 | 2, false, true), + ] { + let (h, m, f) = setup(Pid::In, UsbOutcome::Nak, 0x4000, 0x4007, 0); + let peer = Function::new(UsbOutcome::Nak); + h.attach(1, peer.clone()).unwrap(); + h.state.lock().r[22] = peer_status; + f.address.store(address, Ordering::SeqCst); + h.advance_frames(1); + assert_eq!(f.calls.load(Ordering::SeqCst) > 0, first_selected, "{name}"); + assert_eq!(peer.calls.load(Ordering::SeqCst) > 0, peer_selected, "{name}"); + if address != 0 { + let peer_calls = peer.calls.load(Ordering::SeqCst); + m.words(0x2000, &[(8 << 16) | 5]); + h.advance_frames(1); + assert!(f.calls.load(Ordering::SeqCst) > 0); + assert_eq!(peer.calls.load(Ordering::SeqCst), peer_calls); + } else if !first_selected { + assert_eq!(m.word(0x3000) >> 28, 5); + assert_eq!((m.word(0x3000) >> 26) & 3, 3); + } + } +} + + +/// A guest USB packet can cross physically nonadjacent DMA pages. +#[test] +fn packet_crosses_nonadjacent_pages() { + let (h, m, _) = setup( + Pid::In, + UsbOutcome::Ack(vec![1, 2, 3, 4, 5, 6, 7, 8]), + 0x4ffc, + 0x9003, + 0, + ); + h.advance_frames(1); + assert_eq!(&m.bytes.lock()[0x4ffc..0x5000], &[1, 2, 3, 4]); + assert_eq!(&m.bytes.lock()[0x9000..0x9004], &[5, 6, 7, 8]); + assert_eq!(m.bytes.lock()[0x5000], 0xa5); + assert_eq!(m.word(0x3004), 0); +} + +/// The frame counter catches up after its host timer was delayed. +#[test] +fn clock_read_catches_up_after_timer_starvation() { + let h = Ohci::new(); + let now = Arc::new(AtomicUsize::new(0)); + let t = now.clone(); + h.set_clock(Arc::new(move || t.load(Ordering::Relaxed) as u64)); + h.start(); + h.bar_write(0, 4, 0x80, 4); + now.store(5_000_000, Ordering::Relaxed); + assert!(h.bar_read(0, 0x3c, 4) >= 5); + h.stop(); + let frame = h.bar_read(0, 0x3c, 4); + h.advance_frames(10); + assert_eq!(h.bar_read(0, 0x3c, 4), frame); +} + +/// The frame count follows the guest's clock, not when the machine's poll or +/// a timer thread last ran: a guest that waits 1 ms after going operational +/// and reads HcFmNumber sees the frame, even if nothing else advanced the +/// controller in between (a slow or descheduled host thread). +#[test] +fn frame_number_follows_the_clock_with_no_poll_or_timer() { + let h = Ohci::new(); + let now = Arc::new(AtomicUsize::new(10_000_000)); + let t = now.clone(); + h.set_clock(Arc::new(move || t.load(Ordering::Relaxed) as u64)); + h.start(); + // Part-way into a frame period when the guest sets operational. + now.store(10_400_000, Ordering::Relaxed); + h.bar_write(0, 4, 0x80, 4); + // Entering Operational increments the frame number at once. + assert_eq!(h.bar_read(0, 0x3c, 4), 1); + // 1.1 ms on the guest's clock: the first boundary, 1 ms after the write, has passed. + now.store(11_500_000, Ordering::Relaxed); + assert_eq!(h.bar_read(0, 0x3c, 4), 2); + // A long guest-time gap is delivered in full at the next access. + now.store(31_500_000, Ordering::Relaxed); + assert_eq!(h.bar_read(0, 0x3c, 4), 22); + // Time that has not passed on the guest's clock delivers nothing. + assert_eq!(h.bar_read(0, 0x3c, 4), 22); + h.stop(); +} + +fn fake_clock_ohci(start_ns: usize) -> (Arc, Arc) { + let h = Ohci::new(); + let now = Arc::new(AtomicUsize::new(start_ns)); + let t = now.clone(); + h.set_clock(Arc::new(move || t.load(Ordering::Relaxed) as u64)); + h.start(); + (h, now) +} + +/// OpenHCI 1.0a sections 6.2.1 and 7.3.3: entering UsbOperational loads +/// HcFmRemaining from the frame interval, increments HcFmNumber, and sends the +/// first SOF one frame later, whatever the phase of the free-running clock. +fn operational_entry_restarts_the_frame(from_hcfs: Option) { + let (h, now) = fake_clock_ohci(10_000_000); + let m = Memory::new(); + h.state().connect_dma(m.clone()); + h.state().config().write(1, 6); + h.bar_write(0, 0x18, 0x1000, 4); + // Not operational: no frames count however much time passes. + if let Some(hcfs) = from_hcfs { + h.bar_write(0, 4, hcfs, 4); + } + now.store(15_000_000, Ordering::Relaxed); + assert_eq!(h.bar_read(0, 0x3c, 4), 0); + now.store(15_400_000, Ordering::Relaxed); + h.bar_write(0, 4, 0x80, 4); + assert_eq!(h.bar_read(0, 0x3c, 4), 1); + assert_eq!(h.bar_read(0, 0x38, 4), 0x2edf); + assert_eq!(h.bar_read(0, 0x0c, 4) & 4, 0, "no SOF at the load"); + assert_eq!(m.bytes.lock()[0x1080], 0xa5, "no HCCA write at the load"); + // Just short of one frame after the write: still nothing. + now.store(16_399_999, Ordering::Relaxed); + assert_eq!(h.bar_read(0, 0x3c, 4), 1); + assert_eq!(h.bar_read(0, 0x0c, 4) & 4, 0); + assert_eq!(m.bytes.lock()[0x1080], 0xa5); + // Exactly one frame after: the first SOF and its HCCA frame number. + now.store(16_400_000, Ordering::Relaxed); + assert_eq!(h.bar_read(0, 0x0c, 4) & 4, 4); + assert_eq!(h.bar_read(0, 0x3c, 4), 2); + assert_eq!(m.bytes.lock()[0x1080], 2); + h.stop(); +} + +#[test] +fn operational_entry_from_reset_restarts_the_frame() { + operational_entry_restarts_the_frame(None); +} + +#[test] +fn operational_entry_from_suspend_restarts_the_frame() { + // HCFS 11b. + operational_entry_restarts_the_frame(Some(0xc0)); +} + +#[test] +fn operational_entry_from_resume_restarts_the_frame() { + // HCFS 01b. + operational_entry_restarts_the_frame(Some(0x40)); +} + +/// The increment on entry carries into the frame-number overflow flag like a +/// frame's own, and HcFmRemaining takes the frame-interval toggle bit too. +#[test] +fn operational_entry_sets_overflow_and_loads_the_interval_toggle() { + for (before, after, overflow) in [(0x7fff, 0x8000, true), (0xffff, 0, true), (0x1234, 0x1235, false)] { + let (h, _now) = fake_clock_ohci(10_000_000); + h.state.lock().r[15] = before; + h.bar_write(0, 0x34, 0x8000_2edf, 4); + h.bar_write(0, 4, 0x80, 4); + assert_eq!(h.bar_read(0, 0x3c, 4), after, "from {before:#x}"); + assert_eq!(h.bar_read(0, 0x0c, 4) & 32 != 0, overflow, "from {before:#x}"); + assert_eq!(h.bar_read(0, 0x38, 4), 0x8000_2edf); + h.stop(); + } +} + +/// An interrupt handler can acknowledge a line while the controller raises it. +#[test] +fn reentrant_irq_acknowledgement_leaves_final_line_clear() { + struct Ack { + hc: Weak, + level: AtomicBool, + once: AtomicBool, + } + impl crate::irq::InterruptController for Ack { + fn set_line(&self, _: u32, level: bool) { + if level && !self.once.swap(true, Ordering::SeqCst) { + self.hc.upgrade().unwrap().bar_write(0, 0xc, RHSC, 4); + } + self.level.store(level, Ordering::SeqCst); + } + } + let h = Ohci::new(); + let ack = Arc::new(Ack { + hc: Arc::downgrade(&h), + level: AtomicBool::new(false), + once: AtomicBool::new(false), + }); + h.state().connect_irq(IrqLine::new(ack.clone(), 0)); + h.attached(); + h.bar_write(0, 0x10, 0x80000040, 4); + h.attach(0, Function::new(UsbOutcome::Nak)).unwrap(); + assert!(!ack.level.load(Ordering::SeqCst)); +} + +/// The guest sees the documented write masks, HCR exemptions, and ownership bit. +#[test] +fn register_masks_preserve_hcr_exempt_fields_and_ownership_status() { + let h = Ohci::new(); + h.bar_write(0, 0x18, 0x123456ff, 4); + assert_eq!(h.bar_read(0, 0x18, 4), 0x12345600); + h.bar_write(0, 0x20, 0x123456ff, 4); + assert_eq!(h.bar_read(0, 0x20, 4), 0x123456f0); + h.bar_write(0, 0x10, u32::MAX, 4); + assert_eq!(h.bar_read(0, 0x10, 4), 0xc000007f); + h.bar_write(0, 8, 8, 4); + assert_eq!(h.bar_read(0, 0xc, 4), 1 << 30); + h.bar_write(0, 0xc, 1 << 31, 4); + assert_eq!(h.bar_read(0, 0xc, 4), 1 << 30); + h.bar_write(0, 0xc, 1 << 30, 4); + assert_eq!(h.bar_read(0, 0xc, 4), 0); + h.bar_write(0, 0x48, u32::MAX, 4); + assert_eq!(h.bar_read(0, 0x48, 4), 0xff001b02); + h.bar_write(0, 4, 0x700, 4); + h.bar_write(0, 8, 1, 4); + assert_eq!(h.bar_read(0, 4, 4), 0x3c0); + assert_eq!(h.bar_read(0, 0x48, 4), 0xff001b02); + assert_eq!(h.bar_read(0, 0x10, 4), 0); + h.bar_write(0, 0x3c, 0xffff, 4); + assert_eq!(h.bar_read(0, 0x3c, 4), 0); + h.bar_write(0, 0x18, 1, 1); + assert_eq!(h.bar_read(0, 0x18, 4), 0); +} + +/// The periodic phase controls how the guest control and bulk lists share service. +#[test] +fn periodic_phase_preserves_control_and_bulk_credit() { + let (h, m, f) = setup(Pid::In, UsbOutcome::Nak, 0x4000, 0x4007, 0); + m.words(0x2000, &[(8 << 16) | (1 << 7), 0x3020, 0x3000, 0]); + m.words(0x2100, &[(8 << 16) | (2 << 7), 0x3120, 0x3100, 0]); + m.words(0x3100, &[0xf0100000, 0x4100, 0x3120, 0x4107]); + m.words(0x2200, &[(8 << 16) | (3 << 7), 0x3220, 0x3200, 0]); + m.words(0x3200, &[0xf0100000, 0x4200, 0x3220, 0x4207]); + for i in 0..32 { + m.words(0x1000 + i * 4, &[0x2200]); + } + let mut s = h.state.lock().clone(); + s.active_fi = 5 * 256 - 1; + s.r[16] = 3 * 256; + s.r[1] = 0xb5; + s.r[10] = 0x2100; + s.r[2] |= 4; + h.schedule(&mut s).unwrap(); + let order: Vec<_> = f.packets.lock().iter().map(|p| p.endpoint).collect(); + assert_eq!(order, [1, 1, 3, 2, 1]); + assert_eq!(s.r[14], 0); +} diff --git a/src/dev/ps2.rs b/src/dev/ps2.rs index fa24f7b6e..068e70084 100644 --- a/src/dev/ps2.rs +++ b/src/dev/ps2.rs @@ -57,8 +57,25 @@ struct Ps2State { const MOUSE_MAX: i32 = 127; /// Combined PS/2 Keyboard and Mouse Controller +/// Another input device on the same host keyboard and mouse: a machine whose +/// guest may use USB instead of PS/2 (the Fuel) fits one beside the 8042 or +/// IOC3 controller, and host input reaches both. Each device drops what its +/// guest has not enabled, as the PS/2 side does. +pub trait HostInputSink: Send + Sync { + fn key(&self, key: KeyCode, pressed: bool); + /// Buttons in PS/2 order (left 1, right 2, middle 4); `dy` is positive + /// down the screen. + fn mouse(&self, buttons: u8, dx: i32, dy: i32, dz: i32); + /// Whether the guest has enabled this mouse. + fn mouse_enabled(&self) -> bool; + /// Mouse movement queued that the guest has not read yet. + fn mouse_pending(&self) -> usize; +} + pub struct Ps2Controller { state: Mutex, + /// Other devices fed from the same host input. + sinks: Mutex>>, /// The controller's interrupt: up while a byte waits to be read. irq: Option, running: AtomicBool, @@ -91,6 +108,7 @@ impl Ps2Controller { held_keys: [false; 256], }), irq, + sinks: Mutex::new(Vec::new()), running: AtomicBool::new(false), layout: AtomicU8::new(KeyboardLayout::Ansi as u8), } @@ -526,17 +544,27 @@ impl Ps2Controller { /// clicks are dropped. pub fn mouse_enabled(&self) -> bool { let state = self.state.lock(); - state.mouse_enabled && state.config & 0x20 == 0 + (state.mouse_enabled && state.config & 0x20 == 0) + || self.sinks.lock().iter().any(|s| s.mouse_enabled()) + } + + /// Feed host keyboard and mouse input to `sink` as well. + pub fn add_sink(&self, sink: Arc) { + self.sinks.lock().push(sink); } /// Mouse bytes queued that the guest has not read yet. pub fn mouse_pending(&self) -> usize { - self.state.lock().mouse_queue_bytes + let own = self.state.lock().mouse_queue_bytes; + self.sinks.lock().iter().fold(own, |n, s| n.max(s.mouse_pending())) } /// Push a scancode byte to the keyboard queue (called from UI) pub fn push_kb(&self, key: KeyCode, pressed: bool) { if !self.running.load(Ordering::Relaxed) { return; } + for sink in self.sinks.lock().iter() { + sink.key(key, pressed); + } let mut state = self.state.lock(); if !state.scanning_enabled || state.config & 0x10 != 0 { return; @@ -968,6 +996,9 @@ impl Ps2Controller { /// - `dz`: scroll wheel delta (positive = scroll up/away from user) pub fn push_mouse_input(&self, buttons: u8, dx: i32, dy: i32, dz: i32) { if !self.running.load(Ordering::Relaxed) { return; } + for sink in self.sinks.lock().iter() { + sink.mouse(buttons, dx, dy, dz); + } let mut state = self.state.lock(); if !state.mouse_enabled || state.config & 0x20 != 0 { return; } @@ -1348,6 +1379,41 @@ impl Saveable for Ps2Controller { mod tests { use super::*; + /// A sink sees host keys and mouse moves even while the guest has not + /// enabled the PS/2 devices, and its mouse counts toward `mouse_enabled` + /// and `mouse_pending`. + #[test] + fn a_sink_shares_the_host_input() { + #[derive(Default)] + struct Sink(Mutex>); + impl HostInputSink for Sink { + fn key(&self, key: KeyCode, pressed: bool) { + self.0.lock().push(format!("{key:?} {pressed}")); + } + fn mouse(&self, buttons: u8, dx: i32, dy: i32, dz: i32) { + self.0.lock().push(format!("mouse {buttons} {dx} {dy} {dz}")); + } + fn mouse_enabled(&self) -> bool { + true + } + fn mouse_pending(&self) -> usize { + 5 + } + } + let p = Ps2Controller::new(None); + let sink = Arc::new(Sink::default()); + p.add_sink(sink.clone()); + p.push_kb(KeyCode::KeyA, true); + assert!(sink.0.lock().is_empty(), "a stopped controller passes nothing on"); + p.start(); + assert!(p.mouse_enabled(), "the sink's mouse is on though the PS/2 mouse is not"); + assert_eq!(p.mouse_pending(), 5); + p.push_kb(KeyCode::KeyA, true); + p.push_mouse_input(2, 3, -4, 1); + assert_eq!(*sink.0.lock(), ["KeyA true", "mouse 2 3 -4 1"]); + assert_eq!(p.pop_from(false), None, "and the PS/2 keyboard, not enabled, took nothing"); + } + /// `type_char` sends what the keys themselves would: Shift around a /// shifted character, Enter for a newline, and nothing for a character /// with no key. diff --git a/src/dev/rad1.rs b/src/dev/rad1.rs index c504e4c22..2d354fc2e 100644 --- a/src/dev/rad1.rs +++ b/src/dev/rad1.rs @@ -69,7 +69,7 @@ use crate::host_audio::{HostOut, Resampler}; use crate::hptimer::{TimerId, TimerManager, TimerReturn}; use crate::snapshot::{field, field_bool, field_u32, field_u32s, field_u64, field_u64_vec, field_u64s, field_u8, StateTable}; use crate::traits::{Device, Saveable}; -use crate::pci::{Bar, PciBusDevice, PciConfig, PciDma}; +use crate::pci::{Bar, PciConfig, PciFunction, PciFunctionState}; pub const PCI_VENDOR_SGI: u16 = 0x10a9; pub const PCI_DEVICE_RAD1: u16 = 0x0005; @@ -815,7 +815,7 @@ impl Rad1 { /// DMA goes down through the Bridge and HEART, which never call back into /// the card, and a catch-up moves at most two revolutions of a ring. pub struct Inner { - pci: PciConfig, + pci: PciFunctionState, state: Mutex, start: Instant, /// When not `i64::MIN`, the clock in nanoseconds, for tests; no timer @@ -837,7 +837,6 @@ pub struct Inner { /// Between `Device::start` and `Device::stop`: timers are armed only /// then. armed: AtomicBool, - dma: OnceLock>, /// `[audio]` settings once host output is enabled; `None` (tests, /// `no_audio`) keeps the output channel running silently. audio_cfg: Mutex>, @@ -860,14 +859,14 @@ fn open_host_output(card: &Inner, cfg: &AudioConfig) -> Option { impl Inner { fn new(me: Weak) -> Self { Self { - pci: PciConfig::new( + pci: PciFunctionState::new(PciConfig::new( PCI_VENDOR_SGI, PCI_DEVICE_RAD1, PCI_CLASS_NOT_DEFINED, RAD1_REVISION, [Bar::Mem(BAR0_SIZE), Bar::Unused, Bar::Unused, Bar::Unused, Bar::Unused, Bar::Unused], 0, - ), + )), state: Mutex::new(State::new(0)), start: Instant::now(), fake_ns: AtomicI64::new(i64::MIN), @@ -878,7 +877,6 @@ impl Inner { tick_pass: Mutex::new(()), timers: OnceLock::new(), armed: AtomicBool::new(false), - dma: OnceLock::new(), audio_cfg: Mutex::new(None), sink: Mutex::new(None), open_host: open_host_output, @@ -1504,7 +1502,7 @@ impl Inner { /// * the flags word, then done last, so the driver never sees a done slot /// with a stale body. No digital input or video reference is connected. fn fill_slot(&self, s: &mut State, slot: u64, now: i64) -> bool { - let Some(dma) = self.dma.get() else { return false }; + let Some(dma) = self.pci.dma() else { return false }; self.sync_all(s, now); let mut mirror = Vec::with_capacity(13 * 4); @@ -1546,7 +1544,7 @@ impl Inner { /// that bit. #[cfg(feature = "perfetto")] fn status_unread(&self, s: &State) -> u32 { - let Some(dma) = self.dma.get() else { return 0 }; + let Some(dma) = self.pci.dma() else { return 0 }; let base = Self::status_base(s); (0..s.status_slots) .filter(|&i| { @@ -1578,7 +1576,7 @@ impl Inner { /// frames are silence. fn chan_move(&self, s: &mut State, idx: usize, mut frames: u64) -> bool { let info = &CHAN_INFO[idx]; - let Some(dma) = self.dma.get() else { return false }; + let Some(dma) = self.pci.dma() else { return false }; // A position read after a long look-away would replay every // revolution of the ring it missed, a DMA per chunk on the CPU // thread. Past one revolution the ring is left as it was found (an @@ -1680,8 +1678,10 @@ impl Device for Rad1 { fn get_clock(&self) -> u64 { 0 } } -impl PciBusDevice for Rad1 { - fn config(&self) -> &PciConfig { +impl PciFunction for Rad1 { + /// The card masters the bus for its status slots and audio rings. It + /// raises no interrupt, so its state never has a line. + fn state(&self) -> &PciFunctionState { &self.inner.pci } @@ -1693,10 +1693,8 @@ impl PciBusDevice for Rad1 { self.inner.write(off, val, width) } - /// The card masters the bus for its status slots and audio rings. It - /// raises no interrupt, so there is no `connect_interrupt`. - fn connect_dma(&self, dma: Arc) { - let _ = self.inner.dma.set(dma); + fn reset(&self) { + self.inner.reset(); } } diff --git a/src/dev/rad1/tests.rs b/src/dev/rad1/tests.rs index 6d286ed02..dae77f9ef 100644 --- a/src/dev/rad1/tests.rs +++ b/src/dev/rad1/tests.rs @@ -4,6 +4,7 @@ use super::*; use crate::hptimer::TimerManager; +use crate::pci::PciDma; use crate::traits::Device; use std::sync::{Arc, Mutex}; @@ -12,8 +13,8 @@ const BAR: u64 = 0x1300_0000; /// A card with BAR0 assigned and memory decoding on, as firmware leaves it. fn card() -> Rad1 { let r = Rad1::new(); - r.config().write(4, BAR as u32); - r.config().write(1, PCI_COMMAND_MEMORY_ON); + r.state().config().write(4, BAR as u32); + r.state().config().write(1, PCI_COMMAND_MEMORY_ON); r.fake_ns.store(0, Ordering::Relaxed); r } @@ -39,7 +40,7 @@ const STARTUP: [(u32, u32); 28] = [ #[test] fn pci_identity() { let r = Rad1::new(); - let c = r.config(); + let c = r.state().config(); assert_eq!(c.read(0), 0x0005_10a9); assert_eq!(c.read(2) & 0xff, 0xc0); assert_eq!(c.read(2), 0xc0); @@ -55,15 +56,14 @@ fn pci_identity() { #[test] fn bar_sizing_and_assignment() { let r = Rad1::new(); - let c = r.config(); + let c = r.state().config(); c.write(4, 0xffff_ffff); assert_eq!(c.read(4), 0xffff_e000); c.write(4, BAR as u32); c.write(1, 2); assert_eq!(c.read(4), BAR as u32); assert_eq!(c.read(1) & 0xffff, 2); - assert_eq!(c.decode(crate::pci::Space::Mem, BAR as u32 + 0x30), Some((0, 0x30))); - assert_eq!(c.decode(crate::pci::Space::Mem, BAR as u32 + 0x2000), None); + assert_eq!(c.bar_regions(), vec![(crate::pci::Space::Mem, 0, BAR as u32, BAR0_SIZE)]); } #[test] @@ -340,7 +340,7 @@ impl PciDma for Ram { fn dma_card() -> (Rad1, Arc) { let r = card(); let ram = Arc::new(Ram(Mutex::new(vec![0xaa; 0x40_0000]))); - r.connect_dma(ram.clone()); + r.state().connect_dma(ram.clone()); (r, ram) } @@ -806,7 +806,7 @@ fn a_long_catch_up_skips_whole_revolutions() { let r = card(); let ram = Arc::new(Ram(Mutex::new(vec![0xaa; 0x40_0000]))); let dma = Arc::new(Counting(ram.clone(), std::sync::atomic::AtomicUsize::new(0))); - r.connect_dma(dma.clone()); + r.state().connect_dma(dma.clone()); program(&r, &RING_INPUT, 0x11_0000, 0xffff_f000); wr(&r, 0x4c, 1); // 1000 s of 44.1 kHz: 44.1 million frames of 8 bytes into a 4 KiB ring. @@ -1195,15 +1195,15 @@ fn slow_host(card: &Inner, _: &crate::config::AudioConfig) -> Option { #[test] fn a_host_slow_to_open_does_not_cost_the_boot_tune_its_start() { let r = Rad1 { inner: Arc::new_cyclic(|me| Inner { open_host: slow_host, ..Inner::new(me.clone()) }) }; - r.config().write(4, BAR as u32); - r.config().write(1, PCI_COMMAND_MEMORY_ON); + r.state().config().write(4, BAR as u32); + r.state().config().write(1, PCI_COMMAND_MEMORY_ON); r.fake_ns.store(0, Ordering::Relaxed); let ram = Arc::new(Ram(Mutex::new(vec![0; 0x40_0000]))); // A recognisable tune: every frame's samples count up from 1. for (f, w) in ram.0.lock().unwrap()[TUNE_BUF as usize..][..TUNE_FRAMES * 8].chunks_exact_mut(4).enumerate() { w.copy_from_slice(&sample_to_word((f / 2 % 30_000 + 1) as i16).to_le_bytes()); } - r.connect_dma(ram); + r.state().connect_dma(ram); r.enable_output(crate::config::AudioConfig::default()); Device::start(&r); prom_tune(&r, TUNE_CTRL); @@ -1297,6 +1297,31 @@ fn a_hold_up_is_more_than_one_interval_late_and_at_most_200ms_is_held_back() { } } +/// A restore into a card that has been in use equals one into a fresh card. +#[test] +fn rad1_load_into_a_dirtied_instance_equals_load_into_a_fresh_one() { + use crate::traits::Saveable; + let src = card(); + wr(&src, 0xa8, 0x1357_9bdf); + wr(&src, 0x80, 0xffe3_cffe); + let v = src.save_state(); + let used = card(); + wr(&used, 0xa8, 0x2468_ace0); + wr(&used, 0x44, 0xa000_0008); + wr(&used, 0x4c0, 0xa000_c000); + { + let mut s = used.state.lock(); + s.chan[CH_OUT].eod = true; + s.out_cg = 2; + s.relay_muted = Some(true); + } + let fresh = card(); + used.load_state(&v).unwrap(); + fresh.load_state(&v).unwrap(); + assert_eq!(used.save_state(), fresh.save_state()); + assert_eq!(used.save_state(), v); +} + /// A saved card loads into a fresh one: its registers come back, and its /// clock reads what it read at the save rather than the restoring card's. #[test] diff --git a/src/dev/usb_hid.rs b/src/dev/usb_hid.rs new file mode 100644 index 000000000..8f417c05b --- /dev/null +++ b/src/dev/usb_hid.rs @@ -0,0 +1,609 @@ +//! USB HID boot-protocol keyboard and mouse: low-speed functions that plug +//! into an OHCI root port and are fed from the host's keyboard and mouse. +//! +//! Written from the public specifications: USB 1.1 (chapter 9 requests and +//! descriptors), Device Class Definition for HID 1.11 (class requests and +//! Appendix B, the boot interface descriptors and report formats) and the HID +//! Usage Tables (keyboard/keypad page). Each device has one configuration, +//! one boot interface and one interrupt-IN endpoint. +//! +//! The control pipe follows USB 1.1 section 8.5.2: a SETUP packet, an optional +//! data stage and a status stage. SET_ADDRESS takes effect once its status +//! stage completes. Interrupt reports queue one per host-side change so a key +//! tapped and released between two polls is still seen twice. Input that +//! arrives before the guest has selected a configuration is dropped, as an +//! unconfigured device would not report it. +//! +//! The data toggle on the wire is not tracked: the controller model keeps +//! each endpoint's toggle and a lost transfer cannot be replayed here. +//! Remote wakeup, the wheel, and the host keyboard layout are not modelled. + +use crate::dev::ohci::{Pid, UsbFunction, UsbOutcome, UsbPacket}; +use crate::dev::ps2::HostInputSink; +use parking_lot::Mutex; +use std::collections::VecDeque; +use std::sync::Arc; +use winit::keyboard::KeyCode; + +/// A vendor and product pair from the pid.codes open test allocation, since +/// the devices are not products of any vendor. +const VENDOR_ID: u16 = 0x1209; +const PRODUCT_ID: u16 = 0x0001; + +// Standard request codes (USB 1.1 table 9-4) and descriptor types (table 9-5). +const GET_STATUS: u8 = 0; +const CLEAR_FEATURE: u8 = 1; +const SET_FEATURE: u8 = 3; +const SET_ADDRESS: u8 = 5; +const GET_DESCRIPTOR: u8 = 6; +const GET_CONFIGURATION: u8 = 8; +const SET_CONFIGURATION: u8 = 9; +const GET_INTERFACE: u8 = 10; +const SET_INTERFACE: u8 = 11; +const DESC_DEVICE: u8 = 1; +const DESC_CONFIGURATION: u8 = 2; +const DESC_STRING: u8 = 3; +const DESC_HID: u8 = 0x21; +const DESC_REPORT: u8 = 0x22; +// HID class requests (HID 1.11 section 7.2). +const GET_REPORT: u8 = 1; +const GET_IDLE: u8 = 2; +const GET_PROTOCOL: u8 = 3; +const SET_REPORT: u8 = 9; +const SET_IDLE: u8 = 10; +const SET_PROTOCOL: u8 = 11; + +/// HID 1.11 appendix B.1: the boot keyboard's report descriptor. Input: a +/// modifier bit per key and six key usages; output: five LED bits. +pub const KEYBOARD_REPORT_DESCRIPTOR: [u8; 63] = [ + 0x05, 0x01, 0x09, 0x06, 0xa1, 0x01, 0x05, 0x07, 0x19, 0xe0, 0x29, 0xe7, 0x15, 0x00, 0x25, 0x01, + 0x75, 0x01, 0x95, 0x08, 0x81, 0x02, 0x95, 0x01, 0x75, 0x08, 0x81, 0x01, 0x95, 0x05, 0x75, 0x01, + 0x05, 0x08, 0x19, 0x01, 0x29, 0x05, 0x91, 0x02, 0x95, 0x01, 0x75, 0x03, 0x91, 0x01, 0x95, 0x06, + 0x75, 0x08, 0x15, 0x00, 0x25, 0x65, 0x05, 0x07, 0x19, 0x00, 0x29, 0x65, 0x81, 0x00, 0xc0, +]; +/// HID 1.11 appendix B.2: the boot mouse's report descriptor. Three buttons +/// and relative X and Y. +pub const MOUSE_REPORT_DESCRIPTOR: [u8; 50] = [ + 0x05, 0x01, 0x09, 0x02, 0xa1, 0x01, 0x09, 0x01, 0xa1, 0x00, 0x05, 0x09, 0x19, 0x01, 0x29, 0x03, + 0x15, 0x00, 0x25, 0x01, 0x95, 0x03, 0x75, 0x01, 0x81, 0x02, 0x95, 0x01, 0x75, 0x05, 0x81, 0x01, + 0x05, 0x01, 0x09, 0x30, 0x09, 0x31, 0x15, 0x81, 0x25, 0x7f, 0x75, 0x08, 0x95, 0x02, 0x81, 0x06, + 0xc0, 0xc0, +]; + +/// Reports held for the guest before the oldest is dropped. +const QUEUE_LIMIT: usize = 64; + +/// What differs between the two devices: the report they send and how host +/// input becomes one. +pub trait Boot: Send { + const PRODUCT: &'static str; + /// bInterfaceProtocol: 1 keyboard, 2 mouse. + const PROTOCOL: u8; + const REPORT_DESCRIPTOR: &'static [u8]; + const MAX_PACKET: u8; + /// The idle rate at reset, in 4 ms units (HID 1.11 section 7.2.4). + const DEFAULT_IDLE: u8; + fn reset(&mut self); + /// The report GET_REPORT returns now. + fn current(&self) -> Vec; + /// The next queued report, if any. + fn next_report(&mut self) -> Option>; + /// An output report; false when the device has none. + fn set_output(&mut self, _data: &[u8]) -> bool { + false + } + /// Host input waiting for the guest, in reports. + fn backlog(&self) -> usize; +} + +enum Stage { + Idle, + /// Data stage toward the host, then an OUT status. + In { data: Vec, sent: usize }, + /// Data stage from the host for a SET_REPORT of `length` bytes. + Out { length: usize, data: Vec }, + /// Status stage of a request with no data toward the host: the next IN + /// answers with an empty packet and the request takes effect. + StatusIn(Effect), +} + +#[derive(Clone, Copy)] +enum Effect { + None, + Address(u8), + Configuration(u8), + Idle(u8), + Protocol(u8), + Output, +} + +struct Inner { + dev: B, + address: u8, + configured: bool, + /// 0 boot, 1 report; a device starts in report protocol. + protocol: u8, + idle: u8, + remote_wakeup: bool, + stage: Stage, + last_report_ms: u64, +} + +pub struct UsbHid { + inner: Mutex>, +} + +impl UsbHid { + fn with(dev: B) -> Arc { + Arc::new(Self { + inner: Mutex::new(Inner { + dev, + address: 0, + configured: false, + protocol: 1, + idle: B::DEFAULT_IDLE, + remote_wakeup: false, + stage: Stage::Idle, + last_report_ms: 0, + }), + }) + } +} + +fn string_descriptor(s: &str) -> Vec { + let units: Vec = s.encode_utf16().flat_map(u16::to_le_bytes).collect(); + let mut d = vec![2 + units.len() as u8, DESC_STRING]; + d.extend(units); + d +} + +impl Inner { + fn device_descriptor() -> Vec { + let mut d = vec![18, DESC_DEVICE, 0x10, 0x01, 0, 0, 0, 8]; + d.extend(VENDOR_ID.to_le_bytes()); + d.extend(PRODUCT_ID.to_le_bytes()); + // bcdDevice, manufacturer and product strings, no serial, one configuration. + d.extend([0x00, 0x01, 1, 2, 0, 1]); + d + } + + fn hid_descriptor() -> Vec { + let n = B::REPORT_DESCRIPTOR.len() as u16; + let mut d = vec![9, DESC_HID, 0x11, 0x01, 0, 1, DESC_REPORT]; + d.extend(n.to_le_bytes()); + d + } + + /// The configuration descriptor followed by its interface, HID and + /// endpoint descriptors, as GET_DESCRIPTOR(configuration) returns them. + fn configuration_descriptor() -> Vec { + let interface = [9, 4, 0, 0, 1, 3, 1, B::PROTOCOL, 0]; + let hid = Self::hid_descriptor(); + // Interrupt IN endpoint 1, polled every 10 ms. + let endpoint = [7, 5, 0x81, 3, B::MAX_PACKET, 0, 10]; + let total = (9 + interface.len() + hid.len() + endpoint.len()) as u16; + let mut d = vec![9, DESC_CONFIGURATION]; + d.extend(total.to_le_bytes()); + // One interface, configuration 1, no string, bus powered with remote + // wakeup, 100 mA. + d.extend([1, 1, 0, 0xa0, 50]); + d.extend(interface); + d.extend(hid); + d.extend(endpoint); + d + } + + fn string(index: u8) -> Option> { + match index { + 0 => Some(vec![4, DESC_STRING, 0x09, 0x04]), + 1 => Some(string_descriptor("iris")), + 2 => Some(string_descriptor(B::PRODUCT)), + _ => None, + } + } + + fn reset(&mut self) { + self.address = 0; + self.configured = false; + self.protocol = 1; + self.idle = B::DEFAULT_IDLE; + self.remote_wakeup = false; + self.stage = Stage::Idle; + self.dev.reset(); + } + + /// Decode a SETUP packet (USB 1.1 table 9-2). None is a request this + /// device does not have, answered with a STALL. + fn setup(&mut self, p: &[u8]) -> Option<()> { + let [rt, req, vl, vh, il, _ih, ll, lh] = <[u8; 8]>::try_from(p).ok()?; + let value = u16::from_le_bytes([vl, vh]); + let length = u16::from_le_bytes([ll, lh]) as usize; + let reply = |data: Vec| Stage::In { sent: 0, data: data[..data.len().min(length)].to_vec() }; + let (kind, recipient) = ((rt >> 5) & 3, rt & 0x1f); + let to_host = rt & 0x80 != 0; + self.stage = match (kind, recipient, to_host, req) { + // Standard requests (USB 1.1 section 9.4). + (0, 0, true, GET_STATUS) => reply(vec![u8::from(self.remote_wakeup) << 1, 0]), + (0, 1 | 2, true, GET_STATUS) if value == 0 => reply(vec![0, 0]), + (0, 0, false, CLEAR_FEATURE) if value == 1 => { + self.remote_wakeup = false; + Stage::StatusIn(Effect::None) + } + (0, 0, false, SET_FEATURE) if value == 1 => { + self.remote_wakeup = true; + Stage::StatusIn(Effect::None) + } + // Clearing a halt on the interrupt endpoint is accepted; nothing halts it. + (0, 2, false, CLEAR_FEATURE) if value == 0 && il & 0x8f == 0x81 => Stage::StatusIn(Effect::None), + (0, 0, false, SET_ADDRESS) if value <= 127 => Stage::StatusIn(Effect::Address(value as u8)), + (0, 0, true, GET_DESCRIPTOR) => match (vh, vl) { + (DESC_DEVICE, 0) => reply(Self::device_descriptor()), + (DESC_CONFIGURATION, 0) => reply(Self::configuration_descriptor()), + (DESC_STRING, n) => reply(Self::string(n)?), + _ => return None, + }, + (0, 0, true, GET_CONFIGURATION) => reply(vec![u8::from(self.configured)]), + (0, 0, false, SET_CONFIGURATION) if value <= 1 => Stage::StatusIn(Effect::Configuration(value as u8)), + (0, 1, true, GET_INTERFACE) if il == 0 => reply(vec![0]), + (0, 1, false, SET_INTERFACE) if il == 0 && value == 0 => Stage::StatusIn(Effect::None), + // The HID descriptor and report descriptor belong to the interface. + (0, 1, true, GET_DESCRIPTOR) if il == 0 => match (vh, vl) { + (DESC_HID, 0) => reply(Self::hid_descriptor()), + (DESC_REPORT, 0) => reply(B::REPORT_DESCRIPTOR.to_vec()), + _ => return None, + }, + // HID class requests (HID 1.11 section 7.2), interface 0. + (1, 1, true, GET_REPORT) if il == 0 && vh == 1 && vl == 0 => reply(self.dev.current()), + (1, 1, false, SET_REPORT) if il == 0 && vh == 2 && vl == 0 && length > 0 => { + Stage::Out { length, data: Vec::new() } + } + (1, 1, true, GET_IDLE) if il == 0 => reply(vec![self.idle]), + (1, 1, false, SET_IDLE) if il == 0 => Stage::StatusIn(Effect::Idle(vh)), + (1, 1, true, GET_PROTOCOL) if il == 0 => reply(vec![self.protocol]), + (1, 1, false, SET_PROTOCOL) if il == 0 && value <= 1 => Stage::StatusIn(Effect::Protocol(value as u8)), + _ => return None, + }; + Some(()) + } + + fn apply(&mut self, effect: Effect, data: &[u8], now_ms: u64) -> bool { + match effect { + Effect::None => {} + Effect::Address(a) => self.address = a, + Effect::Configuration(c) => { + self.configured = c != 0; + self.dev.reset(); + self.last_report_ms = now_ms; + } + Effect::Idle(rate) => self.idle = rate, + Effect::Protocol(p) => self.protocol = p, + Effect::Output => return self.dev.set_output(data), + } + true + } + + fn control(&mut self, p: UsbPacket) -> UsbOutcome { + match p.pid { + Pid::Setup => match self.setup(&p.data) { + Some(()) => UsbOutcome::Ack(Vec::new()), + None => { + self.stage = Stage::Idle; + UsbOutcome::Stall + } + }, + Pid::In => match &mut self.stage { + Stage::In { data, sent } => { + let n = p.capacity.min(data.len() - *sent); + let chunk = data[*sent..*sent + n].to_vec(); + *sent += n; + UsbOutcome::Ack(chunk) + } + Stage::StatusIn(effect) => { + let effect = *effect; + self.stage = Stage::Idle; + if self.apply(effect, &[], p.time_ms) { UsbOutcome::Ack(Vec::new()) } else { UsbOutcome::Stall } + } + _ => UsbOutcome::Stall, + }, + Pid::Out => match &mut self.stage { + // The status stage of a request that sent data to the host. + Stage::In { .. } if p.data.is_empty() => { + self.stage = Stage::Idle; + UsbOutcome::Ack(Vec::new()) + } + Stage::Out { length, data } => { + data.extend(&p.data); + if data.len() >= *length { + let data = std::mem::take(data); + let ok = self.apply(Effect::Output, &data, p.time_ms); + self.stage = if ok { Stage::StatusIn(Effect::None) } else { Stage::Idle }; + if !ok { + return UsbOutcome::Stall; + } + } + UsbOutcome::Ack(Vec::new()) + } + _ => UsbOutcome::Stall, + }, + } + } + + /// Interrupt IN endpoint 1 (USB 1.1 section 5.7): a report when there is + /// one, otherwise NAK. A nonzero idle rate repeats the last state once the + /// period passes without a change. + fn interrupt(&mut self, p: UsbPacket) -> UsbOutcome { + if !self.configured { + return UsbOutcome::Stall; + } + if let Some(r) = self.dev.next_report() { + self.last_report_ms = p.time_ms; + return UsbOutcome::Ack(r); + } + if self.idle != 0 && p.time_ms.saturating_sub(self.last_report_ms) >= u64::from(self.idle) * 4 { + self.last_report_ms = p.time_ms; + return UsbOutcome::Ack(self.dev.current()); + } + UsbOutcome::Nak + } +} + +impl UsbFunction for UsbHid { + fn reset(&self) { + self.inner.lock().reset(); + } + fn address(&self) -> u8 { + self.inner.lock().address + } + fn low_speed(&self) -> bool { + true + } + fn transact(&self, packet: UsbPacket) -> UsbOutcome { + let mut s = self.inner.lock(); + match packet.endpoint { + 0 => s.control(packet), + 1 if packet.pid == Pid::In => s.interrupt(packet), + _ => UsbOutcome::Stall, + } + } +} + +// --------------------------------------------------------------------------- +// Keyboard +// --------------------------------------------------------------------------- + +/// The HID usage (keyboard/keypad page 0x07) of a host key, or None for a +/// key the page has no entry for. +pub fn usage(key: KeyCode) -> Option { + use KeyCode::*; + Some(match key { + KeyA => 0x04, KeyB => 0x05, KeyC => 0x06, KeyD => 0x07, KeyE => 0x08, KeyF => 0x09, + KeyG => 0x0a, KeyH => 0x0b, KeyI => 0x0c, KeyJ => 0x0d, KeyK => 0x0e, KeyL => 0x0f, + KeyM => 0x10, KeyN => 0x11, KeyO => 0x12, KeyP => 0x13, KeyQ => 0x14, KeyR => 0x15, + KeyS => 0x16, KeyT => 0x17, KeyU => 0x18, KeyV => 0x19, KeyW => 0x1a, KeyX => 0x1b, + KeyY => 0x1c, KeyZ => 0x1d, + Digit1 => 0x1e, Digit2 => 0x1f, Digit3 => 0x20, Digit4 => 0x21, Digit5 => 0x22, + Digit6 => 0x23, Digit7 => 0x24, Digit8 => 0x25, Digit9 => 0x26, Digit0 => 0x27, + Enter => 0x28, Escape => 0x29, Backspace => 0x2a, Tab => 0x2b, Space => 0x2c, + Minus => 0x2d, Equal => 0x2e, BracketLeft => 0x2f, BracketRight => 0x30, + Backslash => 0x31, Semicolon => 0x33, Quote => 0x34, Backquote => 0x35, + Comma => 0x36, Period => 0x37, Slash => 0x38, CapsLock => 0x39, + F1 => 0x3a, F2 => 0x3b, F3 => 0x3c, F4 => 0x3d, F5 => 0x3e, F6 => 0x3f, + F7 => 0x40, F8 => 0x41, F9 => 0x42, F10 => 0x43, F11 => 0x44, F12 => 0x45, + PrintScreen => 0x46, ScrollLock => 0x47, Pause => 0x48, Insert => 0x49, + Home => 0x4a, PageUp => 0x4b, Delete => 0x4c, End => 0x4d, PageDown => 0x4e, + ArrowRight => 0x4f, ArrowLeft => 0x50, ArrowDown => 0x51, ArrowUp => 0x52, + NumLock => 0x53, NumpadDivide => 0x54, NumpadMultiply => 0x55, + NumpadSubtract => 0x56, NumpadAdd => 0x57, NumpadEnter => 0x58, + Numpad1 => 0x59, Numpad2 => 0x5a, Numpad3 => 0x5b, Numpad4 => 0x5c, + Numpad5 => 0x5d, Numpad6 => 0x5e, Numpad7 => 0x5f, Numpad8 => 0x60, + Numpad9 => 0x61, Numpad0 => 0x62, NumpadDecimal => 0x63, + IntlBackslash => 0x64, ContextMenu => 0x65, + _ => return None, + }) +} + +/// The bit of a modifier key in the report's first byte (usages 0xe0 to 0xe7). +fn modifier(key: KeyCode) -> Option { + use KeyCode::*; + Some(match key { + ControlLeft => 0x01, ShiftLeft => 0x02, AltLeft => 0x04, SuperLeft => 0x08, + ControlRight => 0x10, ShiftRight => 0x20, AltRight => 0x40, SuperRight => 0x80, + _ => return None, + }) +} + +#[derive(Default)] +pub struct Keyboard { + modifiers: u8, + /// Pressed keys in the order they went down. + keys: Vec, + queue: VecDeque>, + leds: u8, +} + +impl Keyboard { + /// The 8-byte boot report (HID 1.11 B.1): modifiers, reserved, six keys. + /// More than six keys at once is the phantom state, ErrorRollOver in every slot. + fn report(&self) -> Vec { + let mut r = vec![self.modifiers, 0, 0, 0, 0, 0, 0, 0]; + if self.keys.len() > 6 { + r[2..].fill(0x01); + } else { + r[2..2 + self.keys.len()].copy_from_slice(&self.keys); + } + r + } + + fn push(&mut self) { + if self.queue.len() >= QUEUE_LIMIT { + self.queue.pop_front(); + } + let report = self.report(); + self.queue.push_back(report); + } +} + +impl Boot for Keyboard { + const PRODUCT: &'static str = "USB Keyboard"; + const PROTOCOL: u8 = 1; + const REPORT_DESCRIPTOR: &'static [u8] = &KEYBOARD_REPORT_DESCRIPTOR; + const MAX_PACKET: u8 = 8; + // 500 ms, the recommended keyboard default. + const DEFAULT_IDLE: u8 = 125; + fn reset(&mut self) { + *self = Keyboard::default(); + } + fn current(&self) -> Vec { + self.report() + } + fn next_report(&mut self) -> Option> { + self.queue.pop_front() + } + /// The LED output report (HID 1.11 B.1): num lock, caps lock, scroll + /// lock, compose, kana in bits 0 to 4. + fn set_output(&mut self, data: &[u8]) -> bool { + self.leds = data.first().copied().unwrap_or(0) & 0x1f; + true + } + fn backlog(&self) -> usize { + self.queue.len() + } +} + +pub type UsbKeyboard = UsbHid; + +impl UsbKeyboard { + pub fn new() -> Arc { + Self::with(Keyboard::default()) + } + + /// The LEDs the guest last asked for. + pub fn leds(&self) -> u8 { + self.inner.lock().dev.leds + } +} + +impl HostInputSink for UsbKeyboard { + fn key(&self, key: KeyCode, pressed: bool) { + let mut s = self.inner.lock(); + if !s.configured { + return; + } + let k = &mut s.dev; + if let Some(bit) = modifier(key) { + let before = k.modifiers; + if pressed { k.modifiers |= bit } else { k.modifiers &= !bit } + if k.modifiers != before { + k.push(); + } + } else if let Some(u) = usage(key) { + let at = k.keys.iter().position(|&x| x == u); + match (pressed, at) { + (true, None) => k.keys.push(u), + (false, Some(i)) => { + k.keys.remove(i); + } + _ => return, + } + k.push(); + } + } + fn mouse(&self, _: u8, _: i32, _: i32, _: i32) {} + fn mouse_enabled(&self) -> bool { + false + } + fn mouse_pending(&self) -> usize { + 0 + } +} + +// --------------------------------------------------------------------------- +// Mouse +// --------------------------------------------------------------------------- + +#[derive(Default)] +pub struct Mouse { + buttons: u8, + /// Movement not yet in a report. + dx: i32, + dy: i32, + queue: VecDeque>, +} + +impl Mouse { + /// Move accumulated motion into reports, at most a signed byte on each + /// axis apiece (HID 1.11 B.2: buttons, X, Y). `force` sends one even + /// with no motion, for a button change. + fn flush(&mut self, mut force: bool) { + while (force || self.dx != 0 || self.dy != 0) && self.queue.len() < QUEUE_LIMIT { + let (x, y) = (self.dx.clamp(-127, 127), self.dy.clamp(-127, 127)); + self.dx -= x; + self.dy -= y; + self.queue.push_back(vec![self.buttons, x as i8 as u8, y as i8 as u8]); + force = false; + } + } +} + +impl Boot for Mouse { + const PRODUCT: &'static str = "USB Mouse"; + const PROTOCOL: u8 = 2; + const REPORT_DESCRIPTOR: &'static [u8] = &MOUSE_REPORT_DESCRIPTOR; + const MAX_PACKET: u8 = 4; + // A mouse reports only on change. + const DEFAULT_IDLE: u8 = 0; + fn reset(&mut self) { + *self = Mouse::default(); + } + fn current(&self) -> Vec { + vec![self.buttons, 0, 0] + } + fn next_report(&mut self) -> Option> { + if self.queue.is_empty() { + self.flush(false); + } + self.queue.pop_front() + } + fn backlog(&self) -> usize { + self.queue.len() + usize::from(self.dx != 0 || self.dy != 0) + } +} + +pub type UsbMouse = UsbHid; + +impl UsbMouse { + pub fn new() -> Arc { + Self::with(Mouse::default()) + } +} + +impl HostInputSink for UsbMouse { + fn key(&self, _: KeyCode, _: bool) {} + fn mouse(&self, buttons: u8, dx: i32, dy: i32, _dz: i32) { + let mut s = self.inner.lock(); + if !s.configured { + return; + } + let m = &mut s.dev; + m.dx = m.dx.saturating_add(dx); + m.dy = m.dy.saturating_add(dy); + let buttons = buttons & 7; + // Motion up to now travels with the old button state. + if buttons != m.buttons { + m.flush(false); + m.buttons = buttons; + m.flush(true); + } else { + m.flush(false); + } + } + fn mouse_enabled(&self) -> bool { + self.inner.lock().configured + } + fn mouse_pending(&self) -> usize { + self.inner.lock().dev.backlog() + } +} + +#[cfg(test)] +mod tests; diff --git a/src/dev/usb_hid/tests.rs b/src/dev/usb_hid/tests.rs new file mode 100644 index 000000000..393985056 --- /dev/null +++ b/src/dev/usb_hid/tests.rs @@ -0,0 +1,448 @@ +// The cases follow USB 1.1 chapter 9 and HID 1.11: descriptor layout and +// lengths, the control pipe's stages, and the boot report formats. The last +// ones put a device behind the OHCI model to show an interrupt transfer +// returns a report through guest memory. +use super::*; +use crate::dev::ohci::Ohci; +use crate::pci::{PciDma, PciDmaStatus, PciFunction}; +use crate::traits::Device; + +fn packet(endpoint: u8, pid: Pid, capacity: usize, data: &[u8], time_ms: u64) -> UsbPacket { + UsbPacket { endpoint, pid, toggle: false, capacity, data: data.to_vec(), time_ms } +} + +fn ack(o: UsbOutcome) -> Vec { + match o { + UsbOutcome::Ack(b) => b, + other => panic!("expected an ACK, got {other:?}"), + } +} + +/// A control transfer toward the host: SETUP, IN packets of 8 until short, +/// then the OUT status. +fn control_in(f: &dyn UsbFunction, rt: u8, req: u8, value: u16, index: u16, length: u16) -> Result, UsbOutcome> { + let mut setup = vec![rt, req]; + setup.extend(value.to_le_bytes()); + setup.extend(index.to_le_bytes()); + setup.extend(length.to_le_bytes()); + match f.transact(packet(0, Pid::Setup, 8, &setup, 0)) { + UsbOutcome::Ack(_) => {} + other => return Err(other), + } + let mut data = Vec::new(); + loop { + let chunk = ack(f.transact(packet(0, Pid::In, 8, &[], 0))); + let short = chunk.len() < 8; + data.extend(chunk); + if short || data.len() >= length as usize { + break; + } + } + ack(f.transact(packet(0, Pid::Out, 8, &[], 0))); + Ok(data) +} + +/// A control transfer with no data stage: SETUP then the IN status. +fn control_nodata(f: &dyn UsbFunction, rt: u8, req: u8, value: u16, index: u16) -> UsbOutcome { + let mut setup = vec![rt, req]; + setup.extend(value.to_le_bytes()); + setup.extend(index.to_le_bytes()); + setup.extend(0u16.to_le_bytes()); + match f.transact(packet(0, Pid::Setup, 8, &setup, 0)) { + UsbOutcome::Ack(_) => f.transact(packet(0, Pid::In, 8, &[], 0)), + other => other, + } +} + +fn enumerate(f: &dyn UsbFunction, address: u8) { + f.reset(); + assert!(ack(control_nodata(f, 0x00, SET_ADDRESS, address as u16, 0)).is_empty()); + assert_eq!(f.address(), address); + assert!(ack(control_nodata(f, 0x00, SET_CONFIGURATION, 1, 0)).is_empty()); +} + +fn interrupt(f: &dyn UsbFunction, time_ms: u64) -> UsbOutcome { + f.transact(packet(1, Pid::In, 8, &[], time_ms)) +} + +#[test] +fn the_device_descriptor_names_a_low_speed_hid_function() { + for f in [&*UsbKeyboard::new() as &dyn UsbFunction, &*UsbMouse::new()] { + assert!(f.low_speed()); + let d = control_in(f, 0x80, GET_DESCRIPTOR, 0x0100, 0, 64).unwrap(); + assert_eq!(d.len(), 18); + assert_eq!(d[0], 18, "bLength"); + assert_eq!(d[1], 1, "bDescriptorType"); + assert_eq!(u16::from_le_bytes([d[2], d[3]]), 0x0110, "USB 1.1"); + assert_eq!(d[4], 0, "the class is in the interface"); + assert_eq!(d[7], 8, "a low-speed control pipe carries 8 bytes"); + assert_eq!(d[17], 1, "one configuration"); + // The guest may read just the first 8 bytes before addressing. + assert_eq!(control_in(f, 0x80, GET_DESCRIPTOR, 0x0100, 0, 8).unwrap(), d[..8]); + } +} + +/// A configuration, its interface, the HID descriptor and the endpoint come +/// back in one block whose wTotalLength covers them (USB 1.1 9.6.2). +#[test] +fn the_configuration_block_describes_one_boot_interface_and_one_interrupt_endpoint() { + for (f, protocol, report_len, packet_size) in [ + (&*UsbKeyboard::new() as &dyn UsbFunction, 1u8, KEYBOARD_REPORT_DESCRIPTOR.len(), 8u8), + (&*UsbMouse::new(), 2, MOUSE_REPORT_DESCRIPTOR.len(), 4), + ] { + let c = control_in(f, 0x80, GET_DESCRIPTOR, 0x0200, 0, 255).unwrap(); + assert_eq!(c.len(), 34); + assert_eq!(&c[..2], [9, 2]); + assert_eq!(u16::from_le_bytes([c[2], c[3]]) as usize, c.len(), "wTotalLength"); + assert_eq!((c[4], c[5]), (1, 1), "one interface; configuration value 1"); + assert_eq!(&c[9..14], [9, 4, 0, 0, 1], "interface 0, no alternate, one endpoint"); + assert_eq!(&c[14..17], [3, 1, protocol], "HID class, boot subclass, keyboard or mouse"); + assert_eq!(&c[18..20], [9, 0x21], "HID descriptor"); + assert_eq!(u16::from_le_bytes([c[20], c[21]]), 0x0111, "bcdHID 1.11"); + assert_eq!(&c[24..26], [0x22, report_len as u8], "one report descriptor of its length"); + assert_eq!(&c[27..], [7, 5, 0x81, 3, packet_size, 0, 10], "interrupt IN 1, polled at 10 ms"); + // The HID descriptor is also readable on its own through the interface. + assert_eq!(control_in(f, 0x81, GET_DESCRIPTOR, 0x2100, 0, 9).unwrap(), c[18..27]); + } +} + +#[test] +fn the_report_descriptors_are_the_boot_descriptors() { + let k = UsbKeyboard::new(); + let r = control_in(&*k, 0x81, GET_DESCRIPTOR, 0x2200, 0, 255).unwrap(); + assert_eq!(r, KEYBOARD_REPORT_DESCRIPTOR); + // Usage page generic desktop, usage keyboard, application collection. + assert_eq!(&r[..6], [0x05, 0x01, 0x09, 0x06, 0xa1, 0x01]); + assert_eq!(r.last(), Some(&0xc0), "collection ends"); + let m = UsbMouse::new(); + let r = control_in(&*m, 0x81, GET_DESCRIPTOR, 0x2200, 0, 255).unwrap(); + assert_eq!(r, MOUSE_REPORT_DESCRIPTOR); + assert_eq!(&r[..6], [0x05, 0x01, 0x09, 0x02, 0xa1, 0x01]); +} + +#[test] +fn strings_are_utf16_with_a_language_table() { + let k = UsbKeyboard::new(); + assert_eq!(control_in(&*k, 0x80, GET_DESCRIPTOR, 0x0300, 0, 255).unwrap(), [4, 3, 0x09, 0x04]); + let s = control_in(&*k, 0x80, GET_DESCRIPTOR, 0x0302, 0x0409, 255).unwrap(); + assert_eq!(s[0] as usize, s.len()); + let text: Vec = s[2..].chunks(2).map(|c| u16::from_le_bytes([c[0], c[1]])).collect(); + assert_eq!(String::from_utf16(&text).unwrap(), "USB Keyboard"); + assert!(control_in(&*k, 0x80, GET_DESCRIPTOR, 0x0309, 0x0409, 255).is_err(), "no such string"); +} + +/// SET_ADDRESS takes effect after its status stage, not before (USB 1.1 9.4.6). +#[test] +fn set_address_applies_after_the_status_stage() { + let m = UsbMouse::new(); + let mut setup = vec![0x00, SET_ADDRESS, 9, 0, 0, 0, 0, 0]; + assert!(ack(m.transact(packet(0, Pid::Setup, 8, &setup, 0))).is_empty()); + assert_eq!(m.address(), 0, "still the default address during the status stage"); + assert!(ack(m.transact(packet(0, Pid::In, 8, &[], 0))).is_empty()); + assert_eq!(m.address(), 9); + // An address above 127 is not one. + setup[2] = 200; + assert_eq!(m.transact(packet(0, Pid::Setup, 8, &setup, 0)), UsbOutcome::Stall); + // A bus reset returns the device to the default address. + m.reset(); + assert_eq!(m.address(), 0); +} + +#[test] +fn configuration_state_is_readable_and_unknown_requests_stall() { + let k = UsbKeyboard::new(); + assert_eq!(control_in(&*k, 0x80, GET_CONFIGURATION, 0, 0, 1).unwrap(), [0]); + assert!(ack(control_nodata(&*k, 0x00, SET_CONFIGURATION, 1, 0)).is_empty()); + assert_eq!(control_in(&*k, 0x80, GET_CONFIGURATION, 0, 0, 1).unwrap(), [1]); + assert_eq!(control_nodata(&*k, 0x00, SET_CONFIGURATION, 2, 0), UsbOutcome::Stall); + assert_eq!(control_in(&*k, 0x81, GET_INTERFACE, 0, 0, 1).unwrap(), [0]); + assert_eq!(control_in(&*k, 0x80, GET_STATUS, 0, 0, 2).unwrap(), [0, 0]); + // No device qualifier on a low-speed-only device; no such request. + assert!(control_in(&*k, 0x80, GET_DESCRIPTOR, 0x0600, 0, 10).is_err()); + assert_eq!(control_nodata(&*k, 0x00, 0x77, 0, 0), UsbOutcome::Stall); + // A stalled request leaves the pipe usable for the next SETUP. + assert_eq!(control_in(&*k, 0x80, GET_CONFIGURATION, 0, 0, 1).unwrap(), [1]); + // The interrupt endpoint is not there before a configuration is selected. + let fresh = UsbKeyboard::new(); + assert_eq!(interrupt(&*fresh, 0), UsbOutcome::Stall); +} + +#[test] +fn hid_class_requests_set_and_report_idle_and_protocol() { + let k = UsbKeyboard::new(); + // A device starts in report protocol with the keyboard's 500 ms idle. + assert_eq!(control_in(&*k, 0xa1, GET_PROTOCOL, 0, 0, 1).unwrap(), [1]); + assert_eq!(control_in(&*k, 0xa1, GET_IDLE, 0, 0, 1).unwrap(), [125]); + assert!(ack(control_nodata(&*k, 0x21, SET_PROTOCOL, 0, 0)).is_empty()); + assert_eq!(control_in(&*k, 0xa1, GET_PROTOCOL, 0, 0, 1).unwrap(), [0]); + // SET_IDLE carries the duration in the high byte of wValue. + assert!(ack(control_nodata(&*k, 0x21, SET_IDLE, 0, 0)).is_empty()); + assert_eq!(control_in(&*k, 0xa1, GET_IDLE, 0, 0, 1).unwrap(), [0]); + assert_eq!(control_nodata(&*k, 0x21, SET_PROTOCOL, 2, 0), UsbOutcome::Stall); + // A mouse starts with no idle repeat. + let m = UsbMouse::new(); + assert_eq!(control_in(&*m, 0xa1, GET_IDLE, 0, 0, 1).unwrap(), [0]); +} + +#[test] +fn set_report_takes_the_keyboards_leds_and_a_mouse_has_none() { + let k = UsbKeyboard::new(); + let setup = [0x21, SET_REPORT, 0, 2, 0, 0, 1, 0]; + assert!(ack(k.transact(packet(0, Pid::Setup, 8, &setup, 0))).is_empty()); + assert!(ack(k.transact(packet(0, Pid::Out, 8, &[0x1f | 0x80], 0))).is_empty()); + assert!(ack(k.transact(packet(0, Pid::In, 8, &[], 0))).is_empty(), "status stage"); + assert_eq!(k.leds(), 0x1f, "five LED bits"); + let m = UsbMouse::new(); + assert!(ack(m.transact(packet(0, Pid::Setup, 8, &setup, 0))).is_empty()); + assert_eq!(m.transact(packet(0, Pid::Out, 8, &[1], 0)), UsbOutcome::Stall); +} + +#[test] +fn keyboard_reports_carry_modifiers_and_keys_in_press_order() { + let k = UsbKeyboard::new(); + enumerate(&*k, 3); + assert_eq!(interrupt(&*k, 1), UsbOutcome::Nak, "nothing to report yet"); + k.key(KeyCode::ShiftLeft, true); + k.key(KeyCode::KeyA, true); + k.key(KeyCode::KeyB, true); + k.key(KeyCode::KeyA, false); + k.key(KeyCode::ShiftLeft, false); + k.key(KeyCode::KeyB, false); + let reports: Vec> = (0..6).map(|i| ack(interrupt(&*k, i))).collect(); + assert_eq!(reports, [ + vec![0x02, 0, 0, 0, 0, 0, 0, 0], + vec![0x02, 0, 0x04, 0, 0, 0, 0, 0], + vec![0x02, 0, 0x04, 0x05, 0, 0, 0, 0], + vec![0x02, 0, 0x05, 0, 0, 0, 0, 0], + vec![0x00, 0, 0x05, 0, 0, 0, 0, 0], + vec![0x00, 0, 0, 0, 0, 0, 0, 0], + ], "a tap between polls still reports both edges"); + assert_eq!(interrupt(&*k, 7), UsbOutcome::Nak); +} + +#[test] +fn keyboard_usages_follow_the_usage_table() { + assert_eq!(usage(KeyCode::KeyA), Some(0x04)); + assert_eq!(usage(KeyCode::KeyZ), Some(0x1d)); + assert_eq!(usage(KeyCode::Digit1), Some(0x1e)); + assert_eq!(usage(KeyCode::Digit0), Some(0x27)); + assert_eq!(usage(KeyCode::Enter), Some(0x28)); + assert_eq!(usage(KeyCode::Escape), Some(0x29)); + assert_eq!(usage(KeyCode::Space), Some(0x2c)); + assert_eq!(usage(KeyCode::F1), Some(0x3a)); + assert_eq!(usage(KeyCode::F12), Some(0x45)); + assert_eq!(usage(KeyCode::ArrowRight), Some(0x4f)); + assert_eq!(usage(KeyCode::ArrowUp), Some(0x52)); + assert_eq!(usage(KeyCode::Numpad0), Some(0x62)); + assert_eq!(usage(KeyCode::ControlLeft), None, "modifiers have a report bit, not a usage slot"); + assert_eq!(modifier(KeyCode::ControlLeft), Some(0x01)); + assert_eq!(modifier(KeyCode::SuperRight), Some(0x80)); +} + +#[test] +fn a_seventh_key_reports_rollover_and_release_recovers() { + let k = UsbKeyboard::new(); + enumerate(&*k, 1); + let keys = [KeyCode::KeyA, KeyCode::KeyB, KeyCode::KeyC, KeyCode::KeyD, KeyCode::KeyE, KeyCode::KeyF, KeyCode::KeyG]; + for key in keys { + k.key(key, true); + } + let reports: Vec> = (0..7).map(|i| ack(interrupt(&*k, i))).collect(); + assert_eq!(reports[5], [0, 0, 4, 5, 6, 7, 8, 9]); + assert_eq!(reports[6], [0, 0, 1, 1, 1, 1, 1, 1], "ErrorRollOver"); + k.key(KeyCode::KeyG, false); + assert_eq!(ack(interrupt(&*k, 8)), [0, 0, 4, 5, 6, 7, 8, 9]); + // A repeated press of a held key is not a new edge. + k.key(KeyCode::KeyA, true); + assert_eq!(interrupt(&*k, 9), UsbOutcome::Nak); +} + +#[test] +fn input_before_configuration_is_dropped_and_get_report_reads_the_state() { + let k = UsbKeyboard::new(); + k.key(KeyCode::KeyA, true); + enumerate(&*k, 2); + assert_eq!(interrupt(&*k, 1), UsbOutcome::Nak, "the early key never reached the report queue"); + k.key(KeyCode::KeyQ, true); + assert_eq!(control_in(&*k, 0xa1, GET_REPORT, 0x0100, 0, 8).unwrap(), [0, 0, 0x14, 0, 0, 0, 0, 0]); +} + +/// With a nonzero idle rate the keyboard repeats its state when the period +/// passes without a change (HID 1.11 7.2.4), and with zero it stays quiet. +#[test] +fn the_idle_rate_repeats_the_last_report() { + let k = UsbKeyboard::new(); + enumerate(&*k, 2); + k.key(KeyCode::KeyA, true); + assert_eq!(ack(interrupt(&*k, 10)), [0, 0, 4, 0, 0, 0, 0, 0]); + assert_eq!(interrupt(&*k, 509), UsbOutcome::Nak, "500 ms not yet passed"); + assert_eq!(ack(interrupt(&*k, 510)), [0, 0, 4, 0, 0, 0, 0, 0]); + assert!(ack(control_nodata(&*k, 0x21, SET_IDLE, 0, 0)).is_empty()); + assert_eq!(interrupt(&*k, 5000), UsbOutcome::Nak); +} + +#[test] +fn mouse_reports_carry_buttons_and_signed_motion() { + let m = UsbMouse::new(); + enumerate(&*m, 4); + assert!(m.mouse_enabled()); + assert_eq!(interrupt(&*m, 1), UsbOutcome::Nak); + // Host dy is positive down the screen, as HID Y is. + m.mouse(0, 10, -20, 0); + assert_eq!(ack(interrupt(&*m, 2)), [0, 10, (-20i8) as u8]); + // A press travels with its own report, then the release. + m.mouse(1, 0, 0, 0); + m.mouse(0, 0, 0, 0); + assert_eq!(ack(interrupt(&*m, 3)), [1, 0, 0]); + assert_eq!(ack(interrupt(&*m, 4)), [0, 0, 0]); + // Buttons in PS/2 order: right is 2, middle is 4. + m.mouse(2 | 4, 0, 0, 0); + assert_eq!(ack(interrupt(&*m, 5)), [6, 0, 0]); + assert_eq!(control_in(&*m, 0xa1, GET_REPORT, 0x0100, 0, 3).unwrap(), [6, 0, 0]); + // Boot protocol has no wheel byte: the report stays 3 bytes. + m.mouse(6, 0, 0, 3); + assert_eq!(interrupt(&*m, 6), UsbOutcome::Nak); +} + +#[test] +fn large_motion_splits_across_reports_without_loss() { + let m = UsbMouse::new(); + enumerate(&*m, 4); + m.mouse(0, 300, -200, 0); + assert_eq!(m.mouse_pending(), 3); + let mut sum = (0i32, 0i32); + let mut n = 0; + while let UsbOutcome::Ack(r) = interrupt(&*m, n) { + sum.0 += r[1] as i8 as i32; + sum.1 += r[2] as i8 as i32; + assert!(r[1] as i8 != -128 && r[2] as i8 != -128); + n += 1; + } + assert_eq!(sum, (300, -200)); + assert_eq!(m.mouse_pending(), 0); +} + +#[test] +fn a_mouse_before_configuration_is_not_enabled_and_drops_motion() { + let m = UsbMouse::new(); + assert!(!m.mouse_enabled()); + m.mouse(1, 5, 5, 0); + assert_eq!(m.mouse_pending(), 0); + enumerate(&*m, 1); + assert_eq!(interrupt(&*m, 1), UsbOutcome::Nak); +} + +/// Host input arrives through the PS/2 controller's sinks, so the one +/// interface the GUI, VNC and CI use feeds both kinds of device. +#[test] +fn host_input_reaches_usb_devices_through_the_ps2_controller() { + use crate::dev::ps2::Ps2Controller; + let ps2 = Ps2Controller::new(None); + Device::start(&ps2); + let (k, m) = (UsbKeyboard::new(), UsbMouse::new()); + ps2.add_sink(k.clone()); + ps2.add_sink(m.clone()); + enumerate(&*k, 1); + enumerate(&*m, 2); + assert!(ps2.mouse_enabled(), "the USB mouse counts: the PS/2 mouse is still off"); + ps2.push_kb(KeyCode::KeyA, true); + ps2.type_char('b'); + ps2.push_mouse_input(1, 3, 4, 0); + assert_eq!(ps2.mouse_pending(), 2, "a move and a button change wait for the guest"); + assert_eq!(ack(interrupt(&*k, 1)), [0, 0, 4, 0, 0, 0, 0, 0]); + assert_eq!(ack(interrupt(&*k, 2)), [0, 0, 4, 5, 0, 0, 0, 0]); + assert_eq!(ack(interrupt(&*m, 1)), [0, 3, 4]); + assert_eq!(ack(interrupt(&*m, 2)), [1, 0, 0]); +} + +// ---- Through the OHCI model ------------------------------------------------ + +struct Memory(Mutex>); +impl Memory { + fn new() -> Arc { + Arc::new(Self(Mutex::new(vec![0xa5; 0x10000]))) + } + fn words(&self, at: usize, words: &[u32]) { + for (i, w) in words.iter().enumerate() { + self.0.lock()[at + i * 4..at + i * 4 + 4].copy_from_slice(&w.to_le_bytes()); + } + } + fn word(&self, at: usize) -> u32 { + u32::from_le_bytes(self.0.lock()[at..at + 4].try_into().unwrap()) + } +} +impl PciDma for Memory { + fn dma_read(&self, addr: u64, dst: &mut [u8]) -> PciDmaStatus { + match self.0.lock().get(addr as usize..addr as usize + dst.len()) { + Some(s) => { + dst.copy_from_slice(s); + PciDmaStatus::Ok + } + None => PciDmaStatus::MasterAbort, + } + } + fn dma_write(&self, addr: u64, src: &[u8]) -> PciDmaStatus { + match self.0.lock().get_mut(addr as usize..addr as usize + src.len()) { + Some(s) => { + s.copy_from_slice(src); + PciDmaStatus::Ok + } + None => PciDmaStatus::MasterAbort, + } + } +} + +/// An OHCI controller with the device on port 0, reset and enabled, with +/// the periodic list holding one interrupt-IN endpoint at `address`. +fn ohci_with(f: Arc, address: u8) -> (Arc, Arc) { + let h = Ohci::new(); + let m = Memory::new(); + h.state().connect_dma(m.clone()); + h.attach(0, f.clone()).unwrap(); + h.start(); + h.bar_write(0, 0x54, 0x100, 4); + h.bar_write(0, 0x54, 0x10, 4); + h.advance_frames(10); + // The port reset returned the device to its default state: enumerate it. + enumerate(&*f, address); + m.words(0x1000, &[0; 64]); + // Interrupt table: every slot points at the endpoint. ED: address, endpoint 1, + // direction from the TD, low speed, max packet 8; TD queue 0x3000 .. tail 0x3020. + m.words(0x1000, &[0x2000; 32]); + m.words(0x2000, &[u32::from(address) | (1 << 7) | (1 << 13) | (8 << 16), 0x3020, 0x3000, 0]); + // TD: direction IN (2), buffer 0x4000..0x4007, buffer rounding allowed, interrupt delay 0. + m.words(0x3000, &[0xf000_0000 | (2 << 19) | (1 << 18), 0x4000, 0x3020, 0x4007]); + h.state().config().write(1, 6); + h.bar_write(0, 0x18, 0x1000, 4); + h.bar_write(0, 0x04, 0x84, 4); + (h, m) +} + +/// A report waiting in the device is carried by an interrupt TD into guest +/// memory, with the TD retired as a success. +#[test] +fn an_interrupt_transfer_returns_a_keyboard_report_through_ohci() { + let k = UsbKeyboard::new(); + let (h, m) = ohci_with(k.clone(), 5); + k.key(KeyCode::KeyA, true); + h.advance_frames(2); + let report = m.0.lock()[0x4000..0x4009].to_vec(); + // The done queue is published once the delay for a low-priority completion passes. + h.advance_frames(10); + assert_eq!(report, vec![0, 0, 4, 0, 0, 0, 0, 0, 0xa5], "the report, and nothing past its end"); + assert_eq!(m.word(0x3000) >> 28, 0, "completed without error"); + assert_eq!(m.word(0x1084) & !15, 0x3000, "the TD is on the done queue"); +} + +/// With nothing to report the device NAKs and the transfer stays queued. +#[test] +fn an_interrupt_transfer_with_no_report_stays_pending() { + let m = UsbMouse::new(); + let (h, mem) = ohci_with(m.clone(), 5); + h.advance_frames(3); + assert_eq!(mem.word(0x3000) >> 28, 0xf, "not accessed"); + m.mouse(2, 1, 1, 0); + h.advance_frames(2); + assert_eq!(&mem.0.lock()[0x4000..0x4003], [0, 1, 1]); + assert_eq!(mem.word(0x3000) >> 28, 0); +} diff --git a/src/dev/virtio/pci.rs b/src/dev/virtio/pci.rs index 725292f46..c227963bf 100644 --- a/src/dev/virtio/pci.rs +++ b/src/dev/virtio/pci.rs @@ -12,7 +12,7 @@ //! | `0x3000` | notifications, one dword per queue | 2 | //! //! Everything in the BAR is little-endian as the device sees it; the host -//! bridge has already applied its byte-lane rules (see `PciBusDevice`), so a +//! bridge has already applied its byte-lane rules (see `PciFunction`), so a //! guest driver swaps exactly as it does for any other PCI device on that //! machine. //! @@ -22,7 +22,7 @@ use std::sync::Arc; use crate::irq::IrqLine; -use crate::pci::{Bar, PciBusDevice, PciConfig, PciDma, PCI_INTA}; +use crate::pci::{Bar, PciConfig, PciFunction, PciFunctionState, PCI_INTA}; use crate::traits::Resettable; use super::{PciMemory, VirtioCore}; @@ -57,7 +57,7 @@ fn cap(cfg_type: u8, next: u8, len: u8, offset: u32, length: u32, extra: Option< /// A virtio device as a PCI function. pub struct VirtioPci { core: VirtioCore, - pci: PciConfig, + pci: PciFunctionState, sel: parking_lot::Mutex, } @@ -88,7 +88,7 @@ impl VirtioPci { .with_capabilities(16, &caps); // Subsystem: vendor and the virtio device ID, dword 11. pci.write(11, ((id as u32) << 16) | VENDOR_ID as u32); - Self { core, pci, sel: Default::default() } + Self { core, pci: PciFunctionState::new(pci), sel: Default::default() } } pub fn core(&self) -> &VirtioCore { @@ -171,8 +171,8 @@ fn lanes(off: u32, width: usize, word: impl Fn(u32) -> u32) -> u32 { } } -impl PciBusDevice for VirtioPci { - fn config(&self) -> &PciConfig { +impl PciFunction for VirtioPci { + fn state(&self) -> &PciFunctionState { &self.pci } @@ -208,12 +208,14 @@ impl PciBusDevice for VirtioPci { } } - fn connect_interrupt(&self, line: IrqLine) { - self.core.connect_interrupt(line); - } - - fn connect_dma(&self, dma: Arc) { - self.core.connect_dma(Arc::new(PciMemory(dma))); + /// The shared core drives the line and reaches memory through the bus. + fn attached(&self) { + if let Some(dma) = self.pci.dma() { + self.core.connect_dma(Arc::new(PciMemory(dma.clone()))); + } + if let Some(line) = self.pci.irq() { + self.core.connect_interrupt(line.clone()); + } } } @@ -235,7 +237,7 @@ impl crate::traits::Device for VirtioPci { impl Resettable for VirtioPci { fn power_on(&self) { - self.pci.reset(); + self.pci.config().reset(); *self.sel.lock() = Selectors::default(); self.core.power_on(); } @@ -248,7 +250,7 @@ mod tests { use crate::dev::virtio::console::{Console, PortConfig}; use crate::dev::virtio::{VecMemory, Waker, F_VERSION_1}; use crate::irq::LevelProbe; - use crate::pci::PciDmaStatus; + use crate::pci::{PciDma, PciDmaStatus}; struct Dma(Arc); impl PciDma for Dma { @@ -272,15 +274,16 @@ mod tests { let dev = Arc::new(VirtioPci::new(core)); let mem = Arc::new(VecMemory::new(0x2_0000)); let (probe, line) = LevelProbe::line(); - dev.connect_dma(Arc::new(Dma(mem.clone()))); - dev.connect_interrupt(line); + dev.state().connect_dma(Arc::new(Dma(mem.clone()))); + dev.state().connect_irq(line); + dev.attached(); (dev, mem, port, probe) } #[test] fn header_and_capability_list() { let (dev, ..) = rig(); - let c = dev.config(); + let c = dev.state().config(); assert_eq!(c.read(0), (0x1043 << 16) | 0x1af4); assert_eq!(c.read(2) >> 8, 0x07_80_00); assert_eq!(c.read(1) & 0x0010_0000, 0x0010_0000, "capability list present"); diff --git a/src/dev/xbow.rs b/src/dev/xbow.rs index 3f3352254..95a4b0276 100644 --- a/src/dev/xbow.rs +++ b/src/dev/xbow.rs @@ -40,49 +40,49 @@ pub const fn widget_id(rev: u32, part: u32, mfgr: u32) -> u32 { (rev << 28) | (part << 12) | (mfgr << 1) | 1 } -const FIRST_WIDGET: u8 = 8; -const LINKS: usize = 8; +pub(crate) const FIRST_WIDGET: u8 = 8; +pub(crate) const LINKS: usize = 8; /// The register block's span in widget 0's window. pub const REG_SPAN: u32 = 0x0100_0000; -const IDENTIFICATION: u32 = 0x00; -const STATUS: u32 = 0x08; -const ERR_UPPER: u32 = 0x10; -const ERR_LOWER: u32 = 0x18; -const CONTROL: u32 = 0x20; -const PACKET_TIMEOUT: u32 = 0x28; -const INT_UPPER: u32 = 0x30; -const INT_LOWER: u32 = 0x38; -const ERR_CMD: u32 = 0x40; -const LLP_CONTROL: u32 = 0x48; -const STATUS_CLEAR: u32 = 0x50; -const ARB_RELOAD: u32 = 0x58; -const PERF_A: u32 = 0x60; -const PERF_B: u32 = 0x68; -const NIC: u32 = 0x70; +pub(crate) const IDENTIFICATION: u32 = 0x00; +pub(crate) const STATUS: u32 = 0x08; +pub(crate) const ERR_UPPER: u32 = 0x10; +pub(crate) const ERR_LOWER: u32 = 0x18; +pub(crate) const CONTROL: u32 = 0x20; +pub(crate) const PACKET_TIMEOUT: u32 = 0x28; +pub(crate) const INT_UPPER: u32 = 0x30; +pub(crate) const INT_LOWER: u32 = 0x38; +pub(crate) const ERR_CMD: u32 = 0x40; +pub(crate) const LLP_CONTROL: u32 = 0x48; +pub(crate) const STATUS_CLEAR: u32 = 0x50; +pub(crate) const ARB_RELOAD: u32 = 0x58; +pub(crate) const PERF_A: u32 = 0x60; +pub(crate) const PERF_B: u32 = 0x68; +pub(crate) const NIC: u32 = 0x70; /// Revision code 5 and later: widget 0's reset fence, then links 8 to 15's /// at a stride of 8, then the resource lock and its clear. -const FENCE_BASE: u32 = 0x78; -const FENCES: usize = 1 + LINKS; -const FENCE_END: u32 = FENCE_BASE + 8 * FENCES as u32; -const LOCK: u32 = 0xc0; -const LOCK_CLEAR: u32 = 0xc8; +pub(crate) const FENCE_BASE: u32 = 0x78; +pub(crate) const FENCES: usize = 1 + LINKS; +pub(crate) const FENCE_END: u32 = FENCE_BASE + 8 * FENCES as u32; +pub(crate) const LOCK: u32 = 0xc0; +pub(crate) const LOCK_CLEAR: u32 = 0xc8; const FENCE_MASK: u32 = 0xff; /// The first revision code with fences and the lock (Crossbow 2.0 in the /// software tables; an Octane2's code-5 part shows as 1.4 in its inventory). const REVISION_FENCES: u32 = 5; -const LINK_BASE: u32 = 0x100; -const LINK_STRIDE: u32 = 0x40; -const LINK_END: u32 = LINK_BASE + LINK_STRIDE * LINKS as u32; -const LINK_FLUSH: u32 = 0x00; -const LINK_CONTROL: u32 = 0x08; -const LINK_STATUS: u32 = 0x10; -const LINK_ARB_UPPER: u32 = 0x18; -const LINK_ARB_LOWER: u32 = 0x20; -const LINK_STATUS_CLEAR: u32 = 0x28; -const LINK_RESET: u32 = 0x30; -const LINK_AUX_STATUS: u32 = 0x38; +pub(crate) const LINK_BASE: u32 = 0x100; +pub(crate) const LINK_STRIDE: u32 = 0x40; +pub(crate) const LINK_END: u32 = LINK_BASE + LINK_STRIDE * LINKS as u32; +pub(crate) const LINK_FLUSH: u32 = 0x00; +pub(crate) const LINK_CONTROL: u32 = 0x08; +pub(crate) const LINK_STATUS: u32 = 0x10; +pub(crate) const LINK_ARB_UPPER: u32 = 0x18; +pub(crate) const LINK_ARB_LOWER: u32 = 0x20; +pub(crate) const LINK_STATUS_CLEAR: u32 = 0x28; +pub(crate) const LINK_RESET: u32 = 0x30; +pub(crate) const LINK_AUX_STATUS: u32 = 0x38; const STATUS_REQUEST: u32 = 1 << 23; const STATUS_LINK_REQUESTS: u32 = 0xff00_0000; diff --git a/src/dev/xbridge.rs b/src/dev/xbridge.rs new file mode 100644 index 000000000..306261f4b --- /dev/null +++ b/src/dev/xbridge.rs @@ -0,0 +1,1390 @@ +//! Fuel's XBridge revision 2 contains two independent PCI heads and their +//! shared crossbar. Each head has its own 1024-entry ATE, device windows and +//! interrupt state. Revision 2 places aliases differently from the Octane's +//! single-head Bridge, uses different PCI slot data-phase widths, and sends +//! host DMA through widget 0 with source-aware packets. +//! +//! This remains separate from `bridge.rs` because its two-head topology, +//! alias map and host packet path are externally visible parts of this chip. +//! Each head owns a shared `PciBus` for its functions, BAR decode and +//! interrupt pins, and supplies only what is its own: how a processor access +//! becomes configuration cycles and byte order, the ATE and DMA, and where an +//! interrupt pin goes. Whole-page BAR mappings of the device and alias windows +//! decode PCI PIO from the head's own table, without a `Physical` table; the +//! machine's RAM still uses one. +//! +//! # What is approximated here +//! +//! Slot widths, immediate link initialization, synchronous flushes and +//! revision-2 alias placement follow the project's BSD QEMU model, not a +//! silicon measurement. Timing, external SRAM, flash and diagnostic engines +//! are absent. Undefined accesses latch an error and remain observable. +//! +//! Ported from hw/pci-host/sgi-xbridge.c and hw/misc/sgi-xbridge-xbow.c +//! in jamesbraid/qemu-sgi-origami, revision +//! fe128635b6dfe2511f1bd451684cb30007686df1. + +use crate::dev::w1; +use crate::pci::{BarDevice, BarMapping, ByteOrder, CpuWindow, DevFn, IntxRouting, PciBus, PciDma, PciDmaStatus, PciFunction, PciHost, Space}; +use crate::physical::{read_bytes, write_bytes}; +use crate::traits::{BusDevice, BusRead16, BusRead32, BusRead64, BusRead8, Resettable, BUS_OK}; +use parking_lot::{Mutex, RwLock}; +use std::sync::{Arc, Weak}; + +pub const HOST_WIDGET: u8 = 10; +pub const HEAD_E: u8 = 14; +pub const HEAD_F: u8 = 15; +pub const ATE_ENTRIES: usize = 1024; +pub const ATE_MAPPED: usize = 512; + +/// Offsets of the PCI head register words, at eight-byte strides. +pub mod reg { + pub const IDENTIFICATION: usize = 0x0; + pub const STATUS: usize = 0x8; + pub const ERR_UPPER: usize = 0x10; + pub const ERR_LOWER: usize = 0x18; + pub const CONTROL: usize = 0x20; + pub const REQUEST_TIMEOUT: usize = 0x28; + pub const INT_DEST_UPPER: usize = 0x30; + pub const INT_DEST_LOWER: usize = 0x38; + pub const LINK_CONFIG: usize = 0x48; + pub const TEST_PIN_CONTROL: usize = 0x70; + pub const DIRECT_MAP: usize = 0x80; + pub const SSRAM_PARITY_ERR: usize = 0x90; + pub const ARBITRATION: usize = 0xa0; + pub const NIC: usize = 0xb0; + pub const BUS_TIMEOUT: usize = 0xc0; + pub const PCI_TYPE1_CONFIG: usize = 0xc8; + pub const PCI_ERR_LOWER: usize = 0xd8; + pub const INTERRUPT_STATUS: usize = 0x100; + pub const INTERRUPT_ENABLE: usize = 0x108; + pub const INTERRUPT_RESET: usize = 0x110; + pub const INTERRUPT_MODE: usize = 0x118; + pub const INTERRUPT_DEVICE: usize = 0x120; + pub const INTERRUPT_ERROR_HOST: usize = 0x128; + pub const INTERRUPT_HOST_BASE: usize = 0x130; + pub const INTERRUPT_HOST_LAST: usize = 0x168; + pub const INTERRUPT_MULTIPLE: usize = 0x170; + pub const FORCE_INTERRUPT_BASE: usize = 0x180; + pub const FORCE_PIN_BASE: usize = 0x1c0; + pub const FORCE_PIN_LAST: usize = 0x1f8; + pub const DEVICE_BASE: usize = 0x200; + pub const DEVICE_LAST: usize = 0x238; + pub const RRB_EVEN: usize = 0x280; + pub const RRB_ODD: usize = 0x288; + pub const RRB_CLEAR: usize = 0x298; + pub const ATE_UPPER_BASE: u64 = 0x10000; + pub const ATE_LOWER_BASE: u64 = 0x12000; + pub const ATE_END: u64 = 0x14000; + +} + +/// The crossbar has the Crossbow's widget 0 register map, which `xbow.rs` +/// owns; the register offsets here are its constants. The access rules and +/// reset values below are this chip's own (the Octane's Crossbow model differs +/// in status, link state, wide accesses and where its interrupts go). +use crate::dev::xbow; + +/// Fences run from widget 0's to link 15's. +const FENCE_LAST: u32 = xbow::FENCE_END - 8; +/// Read-only registers between the resource lock and the first link. +const READ_ONLY_BASE: u32 = xbow::LOCK_CLEAR + 8; +const READ_ONLY_LAST: u32 = xbow::LINK_BASE - 8; +fn ix(off: u32) -> usize { + (off / 8) as usize +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct InterruptPacket { + pub source_widget: u8, + pub widget: u8, + pub address: u64, + pub vector: u8, + pub level: bool, +} +pub type InterruptSender = Arc bool + Send + Sync>; + +/// Source identity and placement hints retained for a host's inbound packet path. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct DmaPacket { + pub source_widget: u8, + pub slot: u8, + pub widget: u8, + pub address: u64, + pub write: bool, + pub half_write_requested: bool, + pub half_read_requested: bool, +} +/// Bytes are in XIO address order after bridge placement. A false return +/// refuses that access. The host applies the requested exchange of doubleword halves. +pub type DmaSender = Arc bool + Send + Sync>; + +struct Routes { + targets: RwLock<[Option>; 16]>, + interrupts: RwLock>, + dma: RwLock>, +} +impl Routes { + fn interrupt(&self, p: InterruptPacket) -> bool { + let sender = self.interrupts.read().clone(); + let accepted = if let Some(send) = sender { + send(p) + } else { + let target = self.targets.read()[p.widget as usize].clone(); + target.is_some_and(|t| { + t.write64( + p.address, + p.vector as u64 | if p.level { 1 << 8 } else { 0 }, + ) == BUS_OK + }) + }; + if !accepted { + crate::dlog!(crate::devlog::LogModule::Bedrock, "xbridge: refused interrupt {p:?}"); + } + accepted + } +} + +/// One chip owns both heads and widget 0; views share routing and reset state. +pub struct XBridge { + routes: Arc, + heads: [Arc; 2], + xbow: Arc, +} +impl XBridge { + pub fn new(memory: Arc) -> Arc { + let routes = Arc::new(Routes { + targets: RwLock::new(std::array::from_fn(|i| { + if i == HOST_WIDGET as usize { + Some(memory.clone()) + } else { + None + } + })), + interrupts: RwLock::new(None), + dma: RwLock::new(None), + }); + let heads = [ + XBridgeHead::new(HEAD_E, 1, routes.clone()), + XBridgeHead::new(HEAD_F, 0, routes.clone()), + ]; + let xbow = Arc::new(Crossbar { + st: Mutex::new(XbowState::new()), + order: Mutex::new(()), + routes: routes.clone(), + heads: heads.each_ref().map(Arc::downgrade), + reset_output: RwLock::new(None), + }); + Arc::new(Self { + routes, + heads, + xbow, + }) + } + pub fn register_locks(self: &Arc) { + use crate::locks::register_lock_fn; + for head in &self.heads { + for (what, probe) in [ + ("state", (|h: &XBridgeHead| h.st.is_locked()) as fn(&XBridgeHead) -> bool), + ("irq_order", |h| h.order.is_locked()), + ("microlan", |h| h.microlan.is_locked()), + ] { + let h = Arc::downgrade(head); + let name = format!("xbridge::head{:x}::{what}", head.widget); + register_lock_fn(name, move || h.upgrade().is_some_and(|h| probe(&h))); + } + } + for (what, probe) in [ + ("state", (|x: &Crossbar| x.st.is_locked()) as fn(&Crossbar) -> bool), + ("irq_order", |x| x.order.is_locked()), + ] { + let x = Arc::downgrade(&self.xbow); + register_lock_fn(format!("xbridge::crossbar::{what}"), move || { + x.upgrade().is_some_and(|x| probe(&x)) + }); + } + } + fn pci_head(&self, widget: u8) -> &Arc { + assert!((HEAD_E..=HEAD_F).contains(&widget)); + &self.heads[(widget - HEAD_E) as usize] + } + pub fn head(&self, widget: u8) -> Arc { + self.pci_head(widget).clone() + } + pub fn crossbar(&self) -> Arc { + self.xbow.clone() + } + /// A reader-specific widget-0 view; the host reads port 10 by default. + pub fn crossbar_for_port(&self, port: u8) -> Arc { + assert!((8..=15).contains(&port)); + Arc::new(CrossbarView { + xbow: self.xbow.clone(), + port, + }) + } + /// Host-port processor projection uses the XBridge widget-0 alias. + pub fn widget(&self, widget: u8) -> Option> { + match widget { + 0 | HOST_WIDGET => Some(self.crossbar()), + HEAD_E | HEAD_F => Some(self.head(widget)), + _ => self + .routes + .targets + .read() + .get(widget as usize) + .and_then(Clone::clone), + } + } + /// The PCI bus behind head `widget`: functions are fitted with + /// [`PciBus::attach`] and the SuperIO interrupt is `bus.board_irq(BoardIrq(0))`. + pub fn bus(&self, widget: u8) -> &PciBus { + &self.pci_head(widget).bus + } + pub fn dma(&self, widget: u8, slot: u8) -> Arc { + assert!(slot < 8); + self.pci_head(widget).dma_for(slot) + } + pub fn set_interrupt_sender(&self, send: InterruptSender) { + *self.routes.interrupts.write() = Some(send); + } + pub fn set_dma_sender(&self, send: DmaSender) { + *self.routes.dma.write() = Some(send); + } + pub fn set_dma_64bit(&self, widget: u8, slot: u8, wide: bool) { + assert!(slot < 8); + let head = self.pci_head(widget); + { + let mut st = head.st.lock(); + st.dma_64bit = (st.dma_64bit & !(1 << slot)) | ((wide as u8) << slot); + } + // A 64-bit slot changes the byte order of the 64-bit alias. + head.bus.set_host_windows(&|| head.windows()); + } + pub fn set_wide_slot_mask(&self, widget: u8, mask: u8) { + self.pci_head(widget).st.lock().wide_slots = mask; + } + pub fn set_microlan(&self, widget: u8, bus: w1::Bus) { + *self.pci_head(widget).microlan.lock() = Some(bus); + } + /// An interrupt packet from the widget attached at `p.source_widget`, + /// routed as the heads' packets are. False when nothing takes it. + pub fn send_interrupt(&self, p: InterruptPacket) -> bool { + assert!((8..14).contains(&p.source_widget) && p.source_widget != HOST_WIDGET); + self.routes.interrupt(p) + } + pub fn attach_widget(&self, widget: u8, target: Arc) { + assert!((8..14).contains(&widget) && widget != HOST_WIDGET); + self.routes.targets.write()[widget as usize] = Some(target); + self.set_link_present(widget, true); + } + pub fn set_link_present(&self, widget: u8, present: bool) { + assert!((8..=15).contains(&widget)); + let mut st = self.xbow.st.lock(); + st.present = (st.present & !(1 << (widget - 8))) | ((present as u8) << (widget - 8)); + } + pub fn set_link_8bit(&self, widget: u8, enabled: bool) { + assert!((8..=15).contains(&widget)); + let mut st = self.xbow.st.lock(); + let bit = 1 << (widget - 8); + st.eight_bit = (st.eight_bit & !bit) | if enabled { bit } else { 0 }; + } + /// A pulse for each reset port, outside chip locks. A head's registers, + /// routed PCI lines and function headers are already reset. The board + /// must reset that port's attached engines through this callback. + pub fn set_reset_output(&self, output: Arc) { + *self.xbow.reset_output.write() = Some(output); + } + /// Time is supplied by the machine's virtual clock, in nanoseconds. + pub fn warm_reset(&self, source_port: u8, now_ns: u64) { + self.xbow.warm_reset(source_port, now_ns); + } + pub fn reset(&self) { + for h in &self.heads { + h.reset(); + } + let mut st = self.xbow.st.lock(); + st.reset(); + st.quiet_until = 0; + } +} +impl Resettable for XBridge { + fn power_on(&self) { + self.reset(); + } +} + +struct HeadState { + reg: [u32; 0x300 / 8], + ate: [u64; ATE_ENTRIES], + pins: u8, + dma_64bit: u8, + wide_slots: u8, +} +impl HeadState { + fn r(&self, off: usize) -> u32 { + self.reg[off / 8] + } + fn set(&mut self, off: usize, value: u32) { + self.reg[off / 8] = value; + } + fn reset(&mut self, speed: u8) { + self.reg.fill(0); + self.ate.fill(0); + self.pins = 0; + for (off, value) in [ + (0, 0x2d002049), + (8, 0x20), + (0x20, 0x0f00320f | ((speed as u32) << 4)), + (0x28, 0xfffff), + (reg::LINK_CONFIG, 0x03ff1810), + (0x80, 0x20000), + (0xa0, 0x20008), + (0xc0, 0x10000), + ] { + self.set(off, value); + } + for slot in 0..8 { + self.set( + 0x200 + slot * 8, + 0x18001000 + | if slot < 2 { + slot as u32 * 2 + } else { + slot as u32 + 2 + }, + ); + } + } + fn packet(&self, pin: Option, level: bool) -> InterruptPacket { + let upper = self.r(reg::INT_DEST_UPPER); + let mut address = ((upper as u64 & 0xffff) << 32) | self.r(reg::INT_DEST_LOWER) as u64; + let vector = if let Some(pin) = pin { + let host = self.r(reg::INTERRUPT_HOST_BASE + pin * 8); + address = (address & !(0x3ff << 38)) | (((host >> 8) as u64 & 0x3ff) << 38); + host as u8 + } else { + self.r(reg::INTERRUPT_ERROR_HOST) as u8 + }; + InterruptPacket { + source_widget: (self.r(reg::CONTROL) & 15) as u8, + widget: ((upper >> 16) & 15) as u8, + address, + vector, + level, + } + } +} +// Lock order: order, then microlan (NIC only), then st. +struct XBridgeHead { + widget: u8, + speed: u8, + st: Mutex, + order: Mutex<()>, + routes: Arc, + microlan: Mutex>, + /// The head's PCI bus: its functions, BAR windows and interrupt pins. + bus: PciBus, + /// Whole-page BAR mappings of the device and alias windows, which the + /// processor reaches without a search of the bus. + pio: PioDecode, + /// This head, for the DMA a function is given: it must not keep the head alive. + me: Weak, +} + +/// A run of the head's address space that one BAR covers whole, and the bus +/// device that answers it. +struct PioRun { + start: u64, + end: u64, + dev: Arc, +} + +/// Whole BAR-page runs are few and carry no RAM or native fault-handler state. +/// Readers clone one endpoint and release the table lock before calling a device. +#[derive(Default)] +struct PioDecode { + runs: RwLock>, +} +impl PioDecode { + fn publish(&self, mut runs: Vec) -> Vec { + runs.sort_unstable_by_key(|run| run.start); + debug_assert!(runs.windows(2).all(|pair| pair[0].end <= pair[1].start)); + std::mem::replace(&mut *self.runs.write(), runs) + } + fn target(&self, address: u64, size: usize) -> Option> { + let runs = self.runs.read(); + let i = runs.partition_point(|run| run.start <= address).checked_sub(1)?; + let run = &runs[i]; + (address.checked_add(size as u64)? <= run.end).then(|| run.dev.clone()) + } +} + +/// The head's PCI interrupt routing. Device `d`'s INTA reaches interrupt pin `d`; the +/// pins are shared with the board's own IRQs (the IOC3's SuperIO interrupt +/// is `BoardIrq(0)`). The Fuel uses devices 1, 4 and 5; the others follow the +/// same rule, and INTB to INTD are not wired. +pub static PCI_ROUTING: IntxRouting = IntxRouting { + devices: 8, + irqs: 8, + intx: &[ + (0, [Some(0), None, None, None]), + (1, [Some(1), None, None, None]), + (2, [Some(2), None, None, None]), + (3, [Some(3), None, None, None]), + (4, [Some(4), None, None, None]), + (5, [Some(5), None, None, None]), + (6, [Some(6), None, None, None]), + (7, [Some(7), None, None, None]), + ], + board_irqs: &[0, 1, 2, 3, 4, 5, 6, 7], +}; + +/// The head's CPU windows, by [`CpuWindow`] index: the eight device windows, +/// then the 32-bit and 64-bit memory aliases and the I/O alias. +const DEVICE_WINDOWS: usize = 8; +const ALIAS_MEM32: usize = 8; +const ALIAS_MEM64: usize = 9; +const ALIAS_IO: usize = 10; +const WINDOWS: usize = 11; + +struct SlotDma { + head: Weak, + slot: u8, +} +impl PciDma for SlotDma { + fn dma_read(&self, address: u64, bytes: &mut [u8]) -> PciDmaStatus { + if let Some(h) = self.head.upgrade() { + h.dma(self.slot, address, bytes, false) + } else { + bytes.fill(0); + PciDmaStatus::MasterAbort + } + } + fn dma_write(&self, address: u64, bytes: &[u8]) -> PciDmaStatus { + self.head.upgrade().map_or(PciDmaStatus::MasterAbort, |h| { + h.dma(self.slot, address, &mut bytes.to_vec(), true) + }) + } +} +/// What the head's bus asks of it: the DMA of a requester, and an interrupt +/// pin's level, as the OR of every device line and board IRQ routed to it. +impl PciHost for XBridgeHead { + fn dma_for(&self, requester: u8) -> Arc { + Arc::new(SlotDma { head: self.me.clone(), slot: requester }) + } + fn set_irq(&self, pin: u8, level: bool) { + self.pin(pin as usize, level); + } +} +impl XBridgeHead { + fn new(widget: u8, speed: u8, routes: Arc) -> Arc { + let mut st = HeadState { + reg: [0; 96], + ate: [0; ATE_ENTRIES], + pins: 0, + dma_64bit: 0, + wide_slots: 0, + }; + st.reset(speed); + let this = Arc::new_cyclic(|me: &Weak| Self { + widget, + speed, + st: Mutex::new(st), + order: Mutex::new(()), + routes, + microlan: Mutex::new(None), + // iris's bus numbers count per bus tree; the heads are separate trees. + bus: PciBus::new((widget - HEAD_E) * 16, &PCI_ROUTING, me.clone()), + pio: PioDecode::default(), + me: me.clone(), + }); + this.bus.set_host_windows(&|| this.windows()); + let head = Arc::downgrade(&this); + this.bus.on_bar_mappings(Box::new(move |runs| { + if let Some(head) = head.upgrade() { + head.publish_mappings(runs); + } + })); + this + } + fn pin(&self, pin: usize, level: bool) { + if pin >= 8 { + return; + } + let _order = self.order.lock(); + let packet = { + let mut st = self.st.lock(); + let bit = 1 << pin; + let old = st.pins & bit != 0; + st.pins = (st.pins & !bit) | if level { bit } else { 0 }; + let pins = st.pins as u32; + let status = (st.r(reg::INTERRUPT_STATUS) & !255) | pins; + st.set(reg::INTERRUPT_STATUS, status); + (old != level + && st.r(reg::INTERRUPT_ENABLE) & bit as u32 != 0 + && (level || st.r(reg::INTERRUPT_MODE) & bit as u32 != 0)) + .then(|| st.packet(Some(pin), level)) + }; + if let Some(packet) = packet { + self.routes.interrupt(packet); + } + } + fn reset(&self) { + { + let _order = self.order.lock(); + self.st.lock().reset(self.speed); + } + // RST#: headers, then functions, which drop their own lines. The + // Device(x) registers are back at reset, so the windows move too. + self.bus.reset(); + self.bus.set_host_windows(&|| self.windows()); + } + fn invalid(&self, address: u64) { + let _order = self.order.lock(); + let packet = { + let mut st = self.st.lock(); + let first = st.r(reg::INTERRUPT_STATUS) & (1 << 24) == 0; + let status = st.r(reg::INTERRUPT_STATUS) | (1 << 24); + st.set(reg::INTERRUPT_STATUS, status); + let multi = st.r(reg::INTERRUPT_MULTIPLE) | (1 << 24); + st.set(reg::INTERRUPT_MULTIPLE, multi); + if first { + st.set(reg::ERR_UPPER, (address >> 32) as u32); + st.set(reg::ERR_LOWER, address as u32); + } + (first && st.r(reg::INTERRUPT_ENABLE) & (1 << 24) != 0).then(|| st.packet(None, true)) + }; + if let Some(p) = packet { + self.routes.interrupt(p); + } + crate::dlog!(crate::devlog::LogModule::Bedrock, + "xbridge: head {} invalid access at {address:#x}", + self.widget + ); + } + fn defined(off: usize) -> bool { + matches!(off, reg::IDENTIFICATION..=reg::TEST_PIN_CONTROL | reg::DIRECT_MAP | reg::SSRAM_PARITY_ERR | reg::ARBITRATION | reg::NIC | reg::BUS_TIMEOUT..=reg::PCI_ERR_LOWER | reg::INTERRUPT_STATUS..=reg::FORCE_PIN_LAST | reg::DEVICE_BASE..=reg::RRB_CLEAR) + } + fn deferred(address: u64) -> bool { + (0x300..0x600).contains(&address) + || (0x30000..0x30008).contains(&address) + || (0xc00000..0x1000000).contains(&address) + } + fn config(&self, address: u64, size: usize, value: Option) -> u64 { + if size == 8 { + let hi = self.config(address, 4, value.map(|v| v >> 32)); + return (hi << 32) | self.config(address + 4, 4, value.map(|v| v & 0xffffffff)); + } + let reg = ((address & 255) as usize) ^ (4 - size); + let function = ((address >> 8) & 7) as u8; + // The type 0 aperture selects a device by address; the type 1 + // aperture takes it, and the bus, from the PCI Type 1 Configuration + // register. Bus 0 there is this head's bus. + let (bus, device) = if address < 0x28000 { + (Some(self.bus.clone()), ((address - 0x20000) >> 12) as u8) + } else { + let sel = self.st.lock().r(reg::PCI_TYPE1_CONFIG); + (self.bus.find_bus(((sel >> 16) & 255) as u8), ((sel >> 11) & 31) as u8) + }; + let at = DevFn { device, function }; + let mask = u32::MAX >> (32 - size * 8); + if let Some(value) = value { + if let Some(bus) = bus { + bus.config_write(at, reg, size, value as u32 & mask); + } + return 0; + } + match bus.and_then(|b| b.config_read(at, reg, size)) { + Some(v) => v as u64, + None => { + let mut st = self.st.lock(); + let multi = st.r(reg::INTERRUPT_MULTIPLE) | (1 << 11); + st.set(reg::INTERRUPT_MULTIPLE, multi); + mask as u64 + } + } + } + fn reg_write(&self, address: u64, value: u32) { + let off = address as usize & !7; + let _order = self.order.lock(); + let mut packets = Vec::new(); + if off == 0xb0 { + let pulse = (value >> 10) & 0x3ff; + let sample = (value >> 2) & 255; + let mut bus = self.microlan.lock(); + if let Some(bus) = bus.as_mut() { + let level = w1::slot::slot(bus, pulse, sample); + self.st.lock().set(reg::NIC, 2 | level as u32); + } else { + crate::dlog!(crate::devlog::LogModule::Bedrock, "xbridge: head {} NIC has no board", self.widget); + } + return; + } + { + let mut st = self.st.lock(); + let masked = match off { + reg::CONTROL => (value & 0x0fe1f20f) | ((self.speed as u32) << 4), + reg::REQUEST_TIMEOUT | reg::INT_DEST_UPPER => value & 0xfffff, + reg::INT_DEST_LOWER | reg::RRB_EVEN | reg::RRB_ODD => value, + reg::LINK_CONFIG => value & 0x03ffffff, + reg::TEST_PIN_CONTROL => value & 0xffffff, + reg::DIRECT_MAP => value & 0xf3ffff, + reg::ARBITRATION => value & 0x3ff7f, + reg::BUS_TIMEOUT => value & 0x1f03ff, + reg::PCI_TYPE1_CONFIG => value & 0xfff800, + reg::INTERRUPT_DEVICE => value & 0xffffff, + reg::INTERRUPT_ERROR_HOST | reg::INTERRUPT_MODE => value & 255, + reg::INTERRUPT_ENABLE => { + let value = value & 0x7fffffff; + let newly = value & !st.r(reg::INTERRUPT_ENABLE) & st.pins as u32; + st.set(off, value); + for pin in 0..8 { + if newly & (1 << pin) != 0 { + packets.push(st.packet(Some(pin), true)); + } + } + value + } + reg::INTERRUPT_RESET => { + let clear = (if value & 2 != 0 { 1 << 30 } else { 0 }) + | (if value & 8 != 0 { 1 << 24 } else { 0 }); + let status = st.r(reg::INTERRUPT_STATUS) & !clear; + st.set(reg::INTERRUPT_STATUS, status); + let multi = if value & 64 != 0 { + 0 + } else { + st.r(reg::INTERRUPT_MULTIPLE) & !clear + }; + st.set(reg::INTERRUPT_MULTIPLE, multi); + return; + } + reg::FORCE_INTERRUPT_BASE..=reg::FORCE_PIN_LAST => { + let pin = (off - reg::FORCE_INTERRUPT_BASE) / 8 & 7; + if st.r(reg::INTERRUPT_ENABLE) & (1 << pin) == 0 { + crate::dlog!(crate::devlog::LogModule::Bedrock, "xbridge: masked force semantics unresolved"); + } else if off < reg::FORCE_PIN_BASE || st.pins & (1 << pin) != 0 { + packets.push(st.packet(Some(pin), true)); + } + 0 + } + reg::INTERRUPT_HOST_BASE..=reg::INTERRUPT_HOST_LAST => value & 0x3ffff, + reg::DEVICE_BASE..=reg::DEVICE_LAST => value & 0x1eefffff, + _ => { + crate::dlog!(crate::devlog::LogModule::Bedrock, + "xbridge: head {} ignored register write {off:#x}", + self.widget + ); + return; + } + }; + if !(reg::FORCE_INTERRUPT_BASE..=reg::FORCE_PIN_LAST).contains(&off) { + st.set(off, masked); + } + } + for p in packets { + self.routes.interrupt(p); + } + drop(_order); + if off == reg::CONTROL || (reg::DEVICE_BASE..=reg::DEVICE_LAST).contains(&off) { + self.bus.set_host_windows(&|| self.windows()); + } + } + + /// The CPU windows, from the Device(x) registers; see [`WINDOWS`]. + fn windows(&self) -> Vec { + let st = self.st.lock(); + (0..WINDOWS) + .map(|kind| { + if kind < DEVICE_WINDOWS { + let (space, base, _) = Self::device(st.r(reg::DEVICE_BASE + kind * 8), kind); + let (cpu, len) = if kind < 2 { + (0x200000 + kind as u64 * 0x200000, 0x200000) + } else { + (0x600000 + (kind as u64 - 2) * 0x100000, 0x100000) + }; + CpuWindow { space, cpu, pci: base << 20, len, mask: u64::MAX } + } else { + let (cpu, len, space) = match kind { + ALIAS_MEM32 => (0x40000000, 0x40000000, Space::Mem), + ALIAS_MEM64 => (0x80000000, 0x40000000, Space::Mem), + _ => (0x100000000, 0x100000000, Space::Io), + }; + CpuWindow { space, cpu, pci: 0, len, mask: u64::MAX } + } + }) + .collect() + } + + /// Device(x) register `dev` of window `slot`: the space it opens onto, + /// its PCI base in megabytes, and whether it is a 64-bit device. + fn device(dev: u32, slot: usize) -> (Space, u32, bool) { + let memory = dev & (1 << 12) != 0; + let mut base = dev & if memory { 0x3ff } else { 0xfff }; + if slot < 2 { + base &= !1; + } + (if memory { Space::Mem } else { Space::Io }, base, dev & (1 << 22) != 0) + } + + /// The byte order of a CPU access to window `kind` reaching function `f`. A device + /// window follows its Device(x) register: the swap bit gives a byte + /// stream, otherwise the device's width gives word values. An alias + /// follows Control's memory or I/O swap bit; the 64-bit memory alias is + /// 64 bits wide for the slots set 64-bit. + fn order(&self, kind: usize, f: &dyn PciFunction) -> ByteOrder { + let st = self.st.lock(); + if kind < DEVICE_WINDOWS { + let dev = st.r(reg::DEVICE_BASE + kind * 8); + return if dev & (1 << 13) != 0 { ByteOrder::Stream } else { ByteOrder::values(if dev & (1 << 22) != 0 { 8 } else { 4 }) }; + } + if st.r(reg::CONTROL) & (1 << if kind == ALIAS_IO { 23 } else { 22 }) != 0 { + return ByteOrder::Stream; + } + let wide = kind == ALIAS_MEM64 && self.slot_of(f).is_some_and(|slot| st.dma_64bit & (1 << slot) != 0); + ByteOrder::values(if wide { 8 } else { 4 }) + } + + /// The device number `f` sits at. + fn slot_of(&self, f: &dyn PciFunction) -> Option { + let same = |g: &Arc| std::ptr::addr_eq(Arc::as_ptr(g), f as *const dyn PciFunction); + (0..DEVICE_WINDOWS as u8).find(|&device| (0..8).any(|function| self.bus.function(DevFn { device, function }).is_some_and(same))) + } + + /// The bus reports the BAR mappings the head's windows hold: serve them + /// from the head's own table. + fn publish_mappings(&self, runs: Vec) { + let runs = runs + .iter() + .map(|r| { + let order = self.order(r.window as usize, &*r.function); + let base = r.cpu.wrapping_sub(r.off as u64); + PioRun { start: r.cpu, end: r.cpu + r.len, dev: Arc::new(BarDevice::new(r.function.clone(), r.bar, base, u64::MAX, order)) } + }) + .collect(); + // Replaced runs may run shutdown callbacks from their destructor. + drop(self.pio.publish(runs)); + } + + /// The window kind, address space and PCI address that head address + /// `address` reaches, for the slow path. + fn pio_target(&self, address: u64) -> Option<(usize, Space, u32)> { + if (0x200000..0xc00000).contains(&address) { + let (slot, off) = if address < 0x600000 { + (((address - 0x200000) >> 21) as usize, address & 0x1fffff) + } else { + (2 + ((address - 0x600000) >> 20) as usize, address & 0xfffff) + }; + let (space, base, _) = Self::device(self.st.lock().r(reg::DEVICE_BASE + slot * 8), slot); + return Some((slot, space, (base << 20).wrapping_add(off as u32))); + } + match address { + 0x40000000..=0x7fffffff => Some((ALIAS_MEM32, Space::Mem, (address - 0x40000000) as u32)), + 0x80000000..=0xbfffffff => Some((ALIAS_MEM64, Space::Mem, (address - 0x80000000) as u32)), + 0x100000000..=0x1ffffffff => Some((ALIAS_IO, Space::Io, (address - 0x100000000) as u32)), + _ => None, + } + } + + /// A processor access that no whole-page run answers: the bus finds the + /// BAR and the window's byte order carries the bytes. Nothing claiming the + /// address reads as ones and drops a write. + fn slow_pio(&self, address: u64, size: usize, value: Option) -> u64 { + let Some((kind, space, pci)) = self.pio_target(address) else { return 0 }; + let Some((f, bar, off)) = self.bus.decode(space, pci) else { + crate::dlog!(crate::devlog::LogModule::Bedrock, + "xbridge: head {} no PCI BAR for {space:?} {pci:#x}", + self.widget + ); + return if value.is_none() { u64::MAX >> (64 - size * 8) } else { 0 }; + }; + let order = self.order(kind, f); + match value { + None => order.read(f, bar, off, size), + Some(v) => { + order.write(f, bar, off, size, v); + 0 + } + } + } + fn access(&self, address: u64, size: usize, value: Option) -> u64 { + let write = value.is_some(); + if (0x20000..0x29000).contains(&address) { + return self.config(address, size, value); + } + if (0x200000..0xc00000).contains(&address) + || (0x40000000..0xc0000000).contains(&address) + || (0x100000000..0x200000000).contains(&address) + { + // An unaligned access can straddle a BAR's final mapped page. + if address & 0xfff > 0x1000 - size as u64 { + return self.slow_pio(address, size, value); + } + let Some(endpoint) = self.pio.target(address, size) else { + return self.slow_pio(address, size, value); + }; + return if let Some(v) = value { + match size { + 1 => { endpoint.write8(address, v as u8); } + 2 => { endpoint.write16(address, v as u16); } + 4 => { endpoint.write32(address, v as u32); } + _ => { endpoint.write64(address, v); } + } + 0 + } else { + match size { + 1 => endpoint.read8(address).data as u64, + 2 => endpoint.read16(address).data as u64, + 4 => endpoint.read32(address).data as u64, + _ => endpoint.read64(address).data, + } + }; + } + if size == 8 && address & 7 == 0 && (reg::ATE_UPPER_BASE..reg::ATE_LOWER_BASE).contains(&address) && write { + self.st.lock().ate[((address - reg::ATE_UPPER_BASE) / 8) as usize] = + value.unwrap() & 0x0000ffffffffffff; + return 0; + } + if size == 4 && address & 7 == 4 { + if !write { + if (reg::ATE_UPPER_BASE..reg::ATE_LOWER_BASE).contains(&address) { + return self.st.lock().ate[((address - reg::ATE_UPPER_BASE) / 8) as usize] >> 32; + } + if (reg::ATE_LOWER_BASE..reg::ATE_END).contains(&address) { + return self.st.lock().ate[((address - reg::ATE_LOWER_BASE) / 8) as usize] as u32 as u64; + } + } + if address < 0x300 && Self::defined(address as usize & !7) { + if let Some(value) = value { + self.reg_write(address, value as u32); + return 0; + } + return self.st.lock().r(address as usize & !7) as u64; + } + } + if Self::deferred(address) { + crate::dlog!(crate::devlog::LogModule::Bedrock, + "xbridge: head {} deferred access at {address:#x}", + self.widget + ); + } else { + self.invalid(address); + } + 0 + } + fn translate( + &self, + slot: u8, + address: u64, + write: bool, + packets: bool, + ) -> Option<(u8, u64, u64, bool)> { + let mut st = self.st.lock(); + let dev = st.r(reg::DEVICE_BASE + slot as usize * 8); + let wide = st.dma_64bit & (1 << slot) != 0; + let device_swap = dev & (1 << 19) != 0; + let (widget, target, swap, direct, page, wide64) = if (0x40000000..0x40800000) + .contains(&address) + || (0x60000000..0x60800000).contains(&address) + { + let shift = if st.r(reg::CONTROL) & (1 << 21) != 0 { 14 } else { 12 }; + let index = ((address & 0x1fffffff) >> shift) as usize; + if index >= ATE_MAPPED || st.ate[index] & 1 == 0 { + let v = st.r(reg::INTERRUPT_STATUS) | (1 << 30); + st.set(reg::INTERRUPT_STATUS, v); + let v = st.r(reg::INTERRUPT_MULTIPLE) | (1 << 30); + st.set(reg::INTERRUPT_MULTIPLE, v); + crate::dlog!( + crate::devlog::LogModule::Bedrock, + "xbridge: invalid page DMA {address:#x}, entry {index}" + ); + return None; + } + let ate = st.ate[index]; + let mask = (1 << shift) - 1; + ( + ((ate >> 8) & 15) as u8, + (ate & 0x0000fffffffff000 & !mask) | (address & mask), + ate & 32 != 0 || address & (1 << 29) != 0, + false, + true, + false, + ) + } else if address >= 0x1000000000000000 { + ( + (address >> 60) as u8, + address & 0x0000ffffffffffff, + address & (1 << 55) != 0, + false, + false, + true, + ) + } else if (0x80000000..=0xffffffff).contains(&address) { + let direct = st.r(reg::DIRECT_MAP); + let mut target = ((direct as u64 & 0x1ffff) << 31) | (address & 0x7fffffff); + if direct & 0x1ffff == 0 && direct & (1 << 17) != 0 { + target += 0x20000000; + } + ( + ((direct >> 20) & 15) as u8, + target, + device_swap, + true, + false, + false, + ) + } else { + crate::dlog!(crate::devlog::LogModule::Bedrock, "xbridge: unsupported DMA {address:#x}"); + return None; + }; + let half = packets + && swap + && st.wide_slots & (1 << slot) != 0 + && if write { + direct || page || (wide64 && device_swap) + } else { + direct || (wide64 && wide && device_swap) || (page && address & (1 << 29) != 0) + }; + let mask = if swap || half { + 0 + } else if wide { + 7 + } else { + 3 + }; + Some((widget, target, mask, half)) + } + // Retain the existing memory path's probe before issuing a chunk's writes. + // The neutral helper alone can commit bytes before reporting a refusal. + fn write_memory(mem: &dyn BusDevice, address: u64, data: &[u8]) -> bool { + if data.is_empty() { + return true; + } + let first = address & !7; + let Some(end) = address + .checked_add(data.len() as u64) + .and_then(|a| a.checked_add(7)) + else { + return false; + }; + let mut probe = vec![0; ((end & !7) - first) as usize]; + read_bytes(mem, first, &mut probe) == BUS_OK && write_bytes(mem, address, data) == BUS_OK + } + fn dma(&self, slot: u8, address: u64, bytes: &mut [u8], write: bool) -> PciDmaStatus { + let sender = self.routes.dma.read().clone(); + let mut good = true; + let mut done = 0; + while done < bytes.len() { + let Some(at) = address.checked_add(done as u64) else { + if !write { + bytes[done..].fill(0); + } + return PciDmaStatus::MasterAbort; + }; + let n = (0x1000 - (at as usize & 0xfff)).min(bytes.len() - done); + if (0x3fff0000..0x40000000).contains(&at) { + if !write { + bytes[done..done + n].fill(0); + } + done += n; + continue; + } + if !write { + bytes[done..done + n].fill(0); + } + let translated = self.translate(slot, at, write, sender.is_some()); + let accepted = if let Some((widget, target, mask, half)) = translated { + if let Some(send) = &sender { + // Packet legs use naturally aligned accesses, retaining source identity. + let mut offset = 0; + let mut ok = true; + while offset < n { + let p = target + offset as u64; + let mut size = 8usize; + while size > n - offset || p & (size as u64 - 1) != 0 { + size /= 2; + } + let mut data = [0u8; 8]; + if write { + for i in 0..size { + data[i] = bytes[done + + offset + + if half { + i + } else { + i ^ (mask as usize & (size - 1)) + }]; + } + } + let placed = if half { + p + } else { + p ^ (mask & !(size as u64 - 1)) + }; + let packet = DmaPacket { + source_widget: (self.st.lock().r(reg::CONTROL) & 15) as u8, + slot, + widget, + address: placed, + write, + half_write_requested: half && write && size == 8, + half_read_requested: half && !write && size == 8, + }; + let part = send(packet, &mut data[..size]); + ok &= part; + if !write { + for i in 0..size { + bytes[done + offset + i] = if part { + data[if half { + i + } else { + i ^ (mask as usize & (size - 1)) + }] + } else { + 0 + }; + } + } + offset += size; + } + ok + } else { + let target_device = self.routes.targets.read()[widget as usize].clone(); + target_device.is_some_and(|mem| { + let low = target & !mask; + let high = (target + n as u64 + mask) & !mask; + let mut data = vec![0; (high - low) as usize]; + if write { + for i in 0..n { + data[(((target + i as u64) ^ mask) - low) as usize] = + bytes[done + i]; + } + if low == target && high == target + n as u64 { + Self::write_memory(&*mem, low, &data) + } else { + let mut ok = true; + for i in 0..n { + let p = (target + i as u64) ^ mask; + ok &= Self::write_memory( + &*mem, + p, + &data[(p - low) as usize..(p - low) as usize + 1], + ); + } + ok + } + } else { + // Preserve the old helper's covering-doubleword probe + // and zero the whole failed physical chunk, even if + // the neutral helper returned part of its contents. + let first = low & !7; + let mut fetched = vec![0; (((high + 7) & !7) - first) as usize]; + let ok = read_bytes(&*mem, first, &mut fetched) == BUS_OK; + for i in 0..n { + bytes[done + i] = if ok { + fetched[(((target + i as u64) ^ mask) - first) as usize] + } else { + 0 + }; + } + ok + } + }) + } + } else { + false + }; + if !accepted { + good = false; + } + done += n; + } + if good { + PciDmaStatus::Ok + } else { + PciDmaStatus::MasterAbort + } + } +} + +struct XbowState { + reg: [u32; 96], + status: u32, + present: u8, + eight_bit: u8, + quiet_until: u64, +} +impl XbowState { + fn new() -> Self { + let mut st = Self { + reg: [0; 96], + status: 0, + present: (1 << 2) | (1 << 6) | (1 << 7), + eight_bit: 0, + quiet_until: 0, + }; + st.reset(); + st + } + fn reset(&mut self) { + self.reg.fill(0); + self.status = 0; + for (off, value) in [ + (xbow::IDENTIFICATION, 0x2d000001), + (xbow::CONTROL, 0x80), + (xbow::PACKET_TIMEOUT, 0xfffff), + (xbow::INT_UPPER, 0x80000), + (xbow::LLP_CONTROL, 0x03ff1810), + (xbow::ARB_RELOAD, 2), + (xbow::NIC, 2), + (xbow::FENCE_BASE, 255), + ] { + self.reg[ix(off)] = value; + } + for i in 0..8 { + self.reg[ix(xbow::FENCE_BASE + 8 + i * 8)] = 255; + self.reg[ix(xbow::LINK_BASE + xbow::LINK_CONTROL + i * xbow::LINK_STRIDE)] = 0x80000; + } + } + fn packet(&self, level: bool) -> InterruptPacket { + let upper = self.reg[ix(xbow::INT_UPPER)]; + InterruptPacket { + source_widget: 0, + widget: ((upper >> 16) & 15) as u8, + address: ((upper as u64 & 0xffff) << 32) | self.reg[ix(xbow::INT_LOWER)] as u64, + vector: (upper >> 24) as u8, + level, + } + } +} +// Lock order: order, then st. Reset outputs run with both released. +struct Crossbar { + st: Mutex, + order: Mutex<()>, + routes: Arc, + heads: [Weak; 2], + reset_output: RwLock>>, +} +impl Crossbar { + fn reset_port(&self, port: u8) { + if (14..=15).contains(&port) { + if let Some(h) = self.heads[(port - 14) as usize].upgrade() { + h.reset(); + } + } + let out = self.reset_output.read().clone(); + if let Some(out) = out { + out(port); + } + } + fn warm_reset(&self, source: u8, now: u64) { + assert!((8..=15).contains(&source)); + let ports = { + let mut st = self.st.lock(); + if now < st.quiet_until { + return; + } + st.quiet_until = now.saturating_add(10240); + if st.reg[ix(xbow::FENCE_BASE)] & (1 << (source - 8)) == 0 { + return; + } + let ports: Vec<_> = (0..8) + .filter(|&i| st.reg[ix(xbow::FENCE_BASE + 8 + i * 8)] & (1 << (source - 8)) != 0) + .map(|i| i as u8 + 8) + .collect(); + st.reset(); + ports + }; + for port in ports { + self.reset_port(port); + } + } + fn access(&self, port: u8, address: u64, size: usize, value: Option) -> u64 { + let _order = self.order.lock(); + let mut packet = None; + let mut reset = None; + let mut access_error = false; + let result = { + let mut st = self.st.lock(); + let off = address as u32 & !7; + if size != 4 || address & 7 != 4 || address >= xbow::LINK_END as u64 || (READ_ONLY_BASE..=READ_ONLY_LAST).contains(&off) { + st.status |= 32; + access_error = true; + if st.reg[ix(xbow::CONTROL)] & 32 != 0 && st.status & (1 << 23) == 0 { + st.status |= 1 << 23; + packet = Some(st.packet(true)); + } + crate::dlog!( + crate::devlog::LogModule::Bedrock, + "xbridge-xbow: invalid access {address:#x} width {size}" + ); + 0 + } else if let Some(v) = value { + let v = v as u32; + if off >= xbow::LINK_BASE { + match (off - xbow::LINK_BASE) % xbow::LINK_STRIDE { + xbow::LINK_CONTROL => st.reg[ix(off)] = v & 0xbf7f01ff, + xbow::LINK_ARB_UPPER | xbow::LINK_ARB_LOWER => st.reg[ix(off)] = v, + xbow::LINK_RESET => reset = Some((8 + (off - xbow::LINK_BASE) / xbow::LINK_STRIDE) as u8), + _ => { + st.status |= 32; + access_error = true; + crate::dlog!( + crate::devlog::LogModule::Bedrock, + "xbridge-xbow: read-only link register offset {off:#x}" + ); + } + } + } else { + match off { + xbow::CONTROL => st.reg[ix(off)] = v & 0xfe, + xbow::PACKET_TIMEOUT => st.reg[ix(off)] = v & 0xfffff, + xbow::INT_UPPER => st.reg[ix(off)] = (v & 0xff07ffff) | 0x80000, + xbow::INT_LOWER => st.reg[ix(off)] = v, + xbow::ERR_CMD => { + st.reg[ix(xbow::ERR_CMD)] = 0; + st.reg[ix(xbow::ERR_UPPER)] = 0; + st.reg[ix(xbow::ERR_LOWER)] = 0; + } + xbow::LLP_CONTROL => st.reg[ix(off)] = v & 0x3ffffff, + xbow::ARB_RELOAD => st.reg[ix(off)] = v & 0x3f, + xbow::PERF_A | xbow::PERF_B => st.reg[ix(off)] = v & 0x700000, + xbow::NIC => { + st.reg[ix(off)] = (v & 0xffffc) | 2; + if v & 0xffc00 != 0 { + crate::dlog!( + crate::devlog::LogModule::Bedrock, + "xbridge-xbow: one-wire pulse unimplemented at register offset 0x70" + ); + } + } + xbow::FENCE_BASE..=FENCE_LAST => st.reg[ix(off)] = v & 255, + xbow::LOCK => { + if st.reg[ix(off)] == 0 { + st.reg[ix(off)] = v & 255; + } + } + xbow::LOCK_CLEAR => st.reg[ix(xbow::LOCK)] = 0, + _ => { + st.status |= 32; + access_error = true; + crate::dlog!( + crate::devlog::LogModule::Bedrock, + "xbridge-xbow: read-only register offset {off:#x}" + ); + } + } + } + 0 + } else if off >= xbow::LINK_BASE { + let link = (off - xbow::LINK_BASE) / xbow::LINK_STRIDE; + let present = st.present & (1 << link) != 0; + match (off - xbow::LINK_BASE) % xbow::LINK_STRIDE { + xbow::LINK_FLUSH => 0, + xbow::LINK_STATUS | xbow::LINK_STATUS_CLEAR => { + if present { + 1 << 31 + } else { + 1 << 5 + } + } + xbow::LINK_AUX_STATUS => { + (if present { 1 << 5 } else { 1 << 6 }) + | (if st.eight_bit & (1 << link) != 0 { + 1 << 4 + } else { + 0 + }) + } + xbow::LINK_RESET => { + st.status |= 32; + access_error = true; + 0 + } + _ => st.reg[ix(off)] as u64, + } + } else { + match off { + xbow::STATUS => (st.status | ((port as u32) << 6)) as u64, + xbow::STATUS_CLEAR => { + let v = st.status | ((port as u32) << 6); + if st.status & (1 << 23) != 0 { + packet = Some(st.packet(false)); + } + st.status &= !((1 << 23) | 0x3f); + v as u64 + } + xbow::LOCK_CLEAR => st.reg[ix(xbow::LOCK)] as u64, + _ => st.reg[ix(off)] as u64, + } + } + }; + if access_error && packet.is_none() { + let mut st = self.st.lock(); + if st.reg[ix(xbow::CONTROL)] & 32 != 0 && st.status & (1 << 23) == 0 { + st.status |= 1 << 23; + packet = Some(st.packet(true)); + } + } + if let Some(p) = packet { + self.routes.interrupt(p); + } + // Reset outputs may call back into the chip, so release ordering too. + drop(_order); + if let Some(port) = reset { + self.reset_port(port); + } + result + } +} +struct CrossbarView { + xbow: Arc, + port: u8, +} + +macro_rules! bus_view { + ($ty:ty, $read:expr) => { + impl BusDevice for $ty { + fn read8(&self, a: u64) -> BusRead8 { + BusRead8::ok(($read)(self, a, 1, None) as u8) + } + fn read16(&self, a: u64) -> BusRead16 { + BusRead16::ok(($read)(self, a, 2, None) as u16) + } + fn read32(&self, a: u64) -> BusRead32 { + BusRead32::ok(($read)(self, a, 4, None) as u32) + } + fn read64(&self, a: u64) -> BusRead64 { + BusRead64::ok(($read)(self, a, 8, None)) + } + fn write8(&self, a: u64, v: u8) -> u32 { + ($read)(self, a, 1, Some(v as u64)); + BUS_OK + } + fn write16(&self, a: u64, v: u16) -> u32 { + ($read)(self, a, 2, Some(v as u64)); + BUS_OK + } + fn write32(&self, a: u64, v: u32) -> u32 { + ($read)(self, a, 4, Some(v as u64)); + BUS_OK + } + fn write64(&self, a: u64, v: u64) -> u32 { + ($read)(self, a, 8, Some(v)); + BUS_OK + } + } + }; +} +bus_view!(XBridgeHead, |s: &XBridgeHead, a, n, v| s.access(a, n, v)); +bus_view!(Crossbar, |s: &Crossbar, a, n, v| s.access( + HOST_WIDGET, + a, + n, + v +)); +bus_view!(CrossbarView, |s: &CrossbarView, a, n, v| s + .xbow + .access(s.port, a, n, v)); + +#[cfg(test)] +mod tests; diff --git a/src/dev/xbridge/tests.rs b/src/dev/xbridge/tests.rs new file mode 100644 index 000000000..78ec74298 --- /dev/null +++ b/src/dev/xbridge/tests.rs @@ -0,0 +1,756 @@ +// Cases follow tests/qtest/sn1-xbridge-test.c (page-dma, interrupt-packets, +// access-errors), sn1-xbridge-chip-test.c (head-dma-reaches-the-host, +// warm-reset-reaches-the-heads) and sn1-xbridge-xbow-test.c (reset-values, +// lock, fences, link-reset-one-shot) in jamesbraid/qemu-sgi-origami. +use super::*; +use crate::pci::BoardIrq; +use crate::traits::BUS_ERR; + +fn at(device: u8) -> DevFn { + DevFn { device, function: 0 } +} +struct Memory(Mutex>); +impl BusDevice for Memory { + fn read32(&self, a: u64) -> BusRead32 { + let mem = self.0.lock(); + if a as usize + 4 > mem.len() { + return BusRead32::err(); + } + BusRead32::ok( + mem[a as usize..a as usize + 4] + .iter() + .fold(0, |v, &b| (v << 8) | b as u32), + ) + } + fn write32(&self, a: u64, v: u32) -> u32 { + let mut mem = self.0.lock(); + if a as usize + 4 > mem.len() { + return BUS_ERR; + } + for i in 0..4 { + mem[a as usize + i] = (v >> (24 - i * 8)) as u8; + } + BUS_OK + } + + fn read64(&self, a: u64) -> BusRead64 { + let mem = self.0.lock(); + if a as usize + 8 > mem.len() { + return BusRead64::err(); + } + BusRead64::ok( + mem[a as usize..a as usize + 8] + .iter() + .fold(0, |v, &b| (v << 8) | b as u64), + ) + } + fn write64(&self, a: u64, v: u64) -> u32 { + let mut mem = self.0.lock(); + if a as usize + 8 > mem.len() { + return BUS_ERR; + } + for i in 0..8 { + mem[a as usize + i] = (v >> (56 - i * 8)) as u8; + } + BUS_OK + } + fn write64_masked(&self, a: u64, v: u64, mask: u64) -> u32 { + let old = self.read64(a); + if old.status != BUS_OK { + return BUS_ERR; + } + self.write64(a, (old.data & !mask) | (v & mask)) + } + + fn read8(&self, a: u64) -> BusRead8 { + match self.0.lock().get(a as usize) { + Some(&value) => BusRead8 { + data: value, + status: BUS_OK, + }, + None => BusRead8 { + data: 0, + status: BUS_ERR, + }, + } + } + fn write8(&self, a: u64, v: u8) -> u32 { + match self.0.lock().get_mut(a as usize) { + Some(b) => { + *b = v; + BUS_OK + } + None => BUS_ERR, + } + } +} +fn chip() -> (Arc, Arc) { + let mem = Arc::new(Memory(Mutex::new(vec![0; 0x8000]))); + (XBridge::new(mem.clone()), mem) +} + +/// The machine lock registry must not extend the lifetime of a chip or attached function. +#[test] +fn lock_registry_does_not_keep_the_chip_or_its_devices_alive() { + let memory = Arc::new(Memory(Mutex::new(vec![0; 0x8000]))); + let memory_ref = Arc::downgrade(&memory); + let chip = XBridge::new(memory); + let chip_ref = Arc::downgrade(&chip); + let heads = chip.heads.each_ref().map(Arc::downgrade); + let crossbar = Arc::downgrade(&chip.xbow); + let card = Card::with_bar(crate::pci::Bar::Mem(0x1000), 0); + let card_ref = Arc::downgrade(&card); + chip.bus(15).attach(at(1), card.clone()); + chip.set_reset_output(Arc::new(move |_| card.state.config().reset())); + chip.head(15).write32(0x21010, 0x02000000); + chip.head(15).write32(0x21004, 2); + assert!(!chip.pci_head(15).pio.runs.read().is_empty()); + chip.register_locks(); + + assert!(heads.iter().all(|head| head.upgrade().is_some())); + assert!(crossbar.upgrade().is_some()); + assert!(card_ref.upgrade().is_some()); + assert!(memory_ref.upgrade().is_some()); + drop(chip); + assert!(chip_ref.upgrade().is_none()); + assert!(heads.iter().all(|head| head.upgrade().is_none())); + assert!(crossbar.upgrade().is_none()); + assert!(card_ref.upgrade().is_none()); + assert!(memory_ref.upgrade().is_none()); +} +/// A DMA packet reports failure without losing the valid prefix already transferred. +#[test] +fn page_dma_keeps_valid_prefix_and_refuses_invalid_suffix() { + physical_chunk_refusal_does_not_publish_a_prefix(); + let (c, mem) = chip(); + let f = c.head(15); + f.write64(reg::ATE_UPPER_BASE as u64, 0x2000 | (10 << 8) | 0x21); + let dma = c.dma(15, 2); + let bytes = [1, 2, 3, 4, 5, 6, 7, 8]; + assert_eq!(dma.dma_write(0x40000ffc, &bytes), PciDmaStatus::MasterAbort); + assert_eq!(&mem.0.lock()[0x2ffc..0x3000], &[1, 2, 3, 4]); + assert_eq!(&mem.0.lock()[0x3000..0x3004], &[0; 4]); + let mut read = [0xff; 8]; + assert_eq!( + dma.dma_read(0x40000ffc, &mut read), + PciDmaStatus::MasterAbort + ); + assert_eq!(read, [1, 2, 3, 4, 0, 0, 0, 0]); + assert_ne!(f.read32((reg::INTERRUPT_STATUS + 4) as u64).data & (1 << 30), 0); + f.write32((reg::INTERRUPT_RESET + 4) as u64, 2); + assert_eq!(f.read32((reg::INTERRUPT_STATUS + 4) as u64).data & (1 << 30), 0); + f.write64((reg::ATE_UPPER_BASE + 800 * 8) as u64, 0xffff123456789abc); + assert_eq!(f.read32((reg::ATE_UPPER_BASE + 4 + 800 * 8) as u64).data, 0x1234); + assert_eq!(f.read32((reg::ATE_LOWER_BASE + 4 + 800 * 8) as u64).data, 0x56789abc); + f.write64((reg::ATE_UPPER_BASE + 512 * 8) as u64, 0x2a21); + f.write32((reg::DIRECT_MAP + 4) as u64, 9 << 20); + for (address, status) in [(0x80002000, PciDmaStatus::MasterAbort), + (0x3fff8000, PciDmaStatus::Ok), (0x40800000, PciDmaStatus::MasterAbort), + (0x40200000, PciDmaStatus::MasterAbort)] { + let mut bytes = [0xff; 4]; + assert_eq!(dma.dma_read(address, &mut bytes), status); + assert_eq!(bytes, [0; 4]); + } +} +// A mapped memory range whose middle word refuses accesses. +struct FaultMemory(Arc); +impl BusDevice for FaultMemory { + fn read8(&self, a: u64) -> BusRead8 { + if (0x2004..0x2008).contains(&a) { + BusRead8::err() + } else { + self.0.read8(a) + } + } + fn write8(&self, a: u64, v: u8) -> u32 { + if (0x2004..0x2008).contains(&a) { + BUS_ERR + } else { + self.0.write8(a, v) + } + } + fn read32(&self, a: u64) -> BusRead32 { + if (0x2004..0x2008).contains(&a) { + BusRead32::err() + } else { + self.0.read32(a) + } + } + fn write32(&self, a: u64, v: u32) -> u32 { + if (0x2004..0x2008).contains(&a) { + BUS_ERR + } else { + self.0.write32(a, v) + } + } +} +fn physical_chunk_refusal_does_not_publish_a_prefix() { + let mem = Arc::new(Memory(Mutex::new(vec![0xaa; 0x8000]))); + let c = XBridge::new(Arc::new(FaultMemory(mem.clone()))); + let f = c.head(15); + f.write32((reg::DIRECT_MAP + 4) as u64, 10 << 20); + f.write32((reg::DEVICE_BASE + 4) as u64, 1 << 19); + let dma = c.dma(15, 0); + let mut bytes = [0xff; 8]; + assert_eq!( + dma.dma_read(0x80002002, &mut bytes), + PciDmaStatus::MasterAbort + ); + assert_eq!( + bytes, [0; 8], + "a failed physical chunk zeroes its readable prefix and suffix" + ); + assert_eq!( + dma.dma_write(0x80002002, &[1; 8]), + PciDmaStatus::MasterAbort + ); + assert_eq!( + &mem.0.lock()[0x2000..0x2010], + &[0xaa; 16], + "the probe prevents writes before or after the refused word" + ); +} +/// Enabling an asserted guest interrupt pin delivers its programmed destination. +#[test] +fn interrupt_keeps_destination_and_enabling_a_live_pin_asserts() { + let (c, _) = chip(); + let f = c.head(15); + let packets = Arc::new(Mutex::new(Vec::new())); + let capture = packets.clone(); + c.set_interrupt_sender(Arc::new(move |p| { + capture.lock().push(p); + true + })); + f.write32((reg::INT_DEST_UPPER + 4) as u64, 0xa0001); + f.write32((reg::INT_DEST_LOWER + 4) as u64, 0x23450080); + f.write32((reg::INTERRUPT_HOST_BASE + 4) as u64, (0x155 << 8) | 0x42); + let wire = c.bus(15).board_irq(BoardIrq(0)); + wire.set(true); + assert!(packets.lock().is_empty()); + f.write32((reg::INTERRUPT_ENABLE + 4) as u64, 1); + f.write32((reg::INTERRUPT_MODE + 4) as u64, 1); + wire.set(false); + let p = packets.lock(); + assert_eq!(p.len(), 2); + assert_eq!(p[0].source_widget, 15); + assert_eq!(p[0].widget, 10); + assert_eq!(p[0].vector, 0x42); + assert_eq!(p[0].address, (0x123450080 & !(0x3ff << 38)) | (0x155 << 38)); + assert!(p[0].level); + assert!(!p[1].level); + drop(p); + // A device's INTA reaches the head's pin of the same number. + let card = Card::new(); + c.bus(15).attach(at(2), card.clone()); + card.state.irq().unwrap().set(true); + assert_eq!(f.read32((reg::INTERRUPT_STATUS + 4) as u64).data & 255, 1 << 2); + // RST# reaches the function, which drops its own line. + c.reset(); + assert_eq!(f.read32((reg::INTERRUPT_STATUS + 4) as u64).data & 255, 0); +} + +/// Each head has a bus of its own: the same device number on the two heads +/// is two functions, configuration reaches only the head addressed, and a +/// function's INTA raises its own head's pin of that number alone. +#[test] +fn each_head_has_its_own_configuration_space_and_interrupt_pins() { + let (c, _) = chip(); + let (first, second) = (Card::new(), Card::new()); + c.bus(HEAD_E).attach(at(3), first.clone()); + c.bus(HEAD_F).attach(at(3), second.clone()); + let (e, f) = (c.head(HEAD_E), c.head(HEAD_F)); + for head in [&e, &f] { + assert_eq!(head.read32(0x23000).data, 0x56781234); + } + assert_eq!(e.write32(0x2303c, 0x5a), BUS_OK); + assert_eq!(first.state.config().read(15) & 0xff, 0x5a); + assert_eq!(second.state.config().read(15) & 0xff, 0, "the other head's function is untouched"); + let pins = |head: &Arc| head.read32((reg::INTERRUPT_STATUS + 4) as u64).data & 255; + second.state.irq().unwrap().set(true); + assert_eq!((pins(&e), pins(&f)), (0, 1 << 3)); + first.state.irq().unwrap().set(true); + assert_eq!((pins(&e), pins(&f)), (1 << 3, 1 << 3)); + second.state.irq().unwrap().set(false); + assert_eq!((pins(&e), pins(&f)), (1 << 3, 0)); + // An unclaimed slot reads as ones and notes the failed cycle. + assert_eq!(f.read32(0x24000).data, u32::MAX); + assert_ne!(f.read32((reg::INTERRUPT_MULTIPLE + 4) as u64).data & (1 << 11), 0); + assert_eq!(e.read32((reg::INTERRUPT_MULTIPLE + 4) as u64).data & (1 << 11), 0); +} + +/// The guest receives host DMA packets with their original widget and address placement. +#[test] +fn host_dma_places_packets_and_rejects_invalid_addresses() { + let (c, mem) = chip(); + let f = c.head(15); + let d = c.dma(15, 0); + f.write32((reg::DIRECT_MAP + 4) as u64, 10 << 20); + f.write64(reg::ATE_UPPER_BASE as u64, 0x3000 | (10 << 8) | 1); + for (wide, address, target, expected) in [ + (false, 0x80002000, 0x2000, [3, 2, 1, 0, 7, 6, 5, 4]), + (true, 0x80002000, 0x2000, [7, 6, 5, 4, 3, 2, 1, 0]), + (true, 0x60000000, 0x3000, [0, 1, 2, 3, 4, 5, 6, 7]), + (true, (10 << 60) | (1 << 55) | 0x2000, 0x2000, [0, 1, 2, 3, 4, 5, 6, 7]), + ] { + c.set_dma_64bit(15, 0, wide); + assert_eq!(d.dma_write(address, &[0, 1, 2, 3, 4, 5, 6, 7]), PciDmaStatus::Ok); + assert_eq!(&mem.0.lock()[target..target + 8], &expected); + } + let records = Arc::new(Mutex::new(Vec::new())); + let out = records.clone(); + c.set_dma_sender(Arc::new(move |p, b| { + out.lock().push((p, b.to_vec())); + true + })); + c.set_wide_slot_mask(15, 1 << 3); + c.set_dma_64bit(15, 3, true); + f.write32((reg::CONTROL + 4) as u64, 8); + f.write32((reg::DIRECT_MAP + 4) as u64, 10 << 20); + f.write32((reg::DEVICE_BASE + 3 * 8 + 4) as u64, 1 << 19); + assert_eq!( + c.dma(15, 3) + .dma_write(0x80002000, &[0, 1, 2, 3, 4, 5, 6, 7]), + PciDmaStatus::Ok + ); + let r = records.lock(); + assert_eq!(r.len(), 1); + assert_eq!(r[0].0.source_widget, 8); + assert_eq!(r[0].0.slot, 3); + assert_eq!(r[0].0.widget, 10); + assert_eq!(r[0].0.address, 0x2000); + assert!(r[0].0.half_write_requested); + assert_eq!(r[0].1, [0, 1, 2, 3, 4, 5, 6, 7]); + drop(r); + c.set_dma_sender(Arc::new(|p, b| { + if p.address >= 0x2008 { + return false; + } + b.fill(0x42); + true + })); + let mut bytes = [0xff; 16]; + assert_eq!( + c.dma(15, 0).dma_read(0x80002000, &mut bytes), + PciDmaStatus::MasterAbort + ); + assert_eq!( + bytes, + [0x42, 0x42, 0x42, 0x42, 0x42, 0x42, 0x42, 0x42, 0, 0, 0, 0, 0, 0, 0, 0] + ); +} +/// A PCI function may inspect XBridge from its link-reset callback. +#[test] +fn link_reset_callback_resets_a_function_and_can_reenter_the_chip() { + use crate::dev::isp1040::{mbox, reg as isp_reg, CTRL_INT_ENABLE, CTRL_RISC_INT_ENABLE, HOST_CMD_SHIFT}; + use crate::dev::isp12160::Isp12160; + + let (c, _) = chip(); + let isp = Arc::new(Isp12160::new(Isp12160::new_bus(), Isp12160::new_bus())); + c.bus(15).attach(at(1), isp.clone()); + let pin1 = |c: &XBridge| c.head(15).read32((reg::INTERRUPT_STATUS + 4) as u64).data & 2 != 0; + isp.state().config().write(1, 6); + isp.state().config().write(5, 0x02000000); + isp.bar_write(1, isp_reg::BUS_CTRL, u32::from(CTRL_INT_ENABLE | CTRL_RISC_INT_ENABLE), 2); + isp.bar_write(1, isp_reg::MAILBOX_BASE, u32::from(mbox::NOP), 2); + isp.bar_write(1, isp_reg::HOST_CMD, 0x5 << HOST_CMD_SHIFT, 2); + assert!(pin1(&c)); + assert_eq!(isp.bar_read(1, isp_reg::MAILBOX_BASE, 2), u32::from(mbox::COMPLETE)); + + let weak = Arc::downgrade(&c); + let reset_isp = isp.clone(); + c.set_reset_output(Arc::new(move |port| { + assert_eq!(port, 15); + let chip = weak.upgrade().unwrap(); + // Callbacks own attached engines and may inspect or rewire the chip. + assert_eq!(chip.head(port).read32((reg::DIRECT_MAP + 4) as u64).data, 0x20000); + assert_eq!(chip.crossbar().read32((xbow::IDENTIFICATION + 4) as u64).data, 0x2d000001); + assert_eq!(reset_isp.state().config().command(), 0); + reset_isp.power_on(); + chip.set_reset_output(Arc::new(|_| {})); + })); + c.head(15).write32((reg::DIRECT_MAP + 4) as u64, 0xa00000); + c.crossbar().write32((xbow::LINK_BASE + 7 * xbow::LINK_STRIDE + xbow::LINK_RESET + 4) as u64, 1); + assert!(!pin1(&c)); + assert_eq!(isp.state().config().bar_base(1), 0); + assert_eq!(isp.bar_read(1, isp_reg::BUS_CTRL, 2), 0); + assert_eq!(isp.bar_read(1, isp_reg::BUS_STATUS, 2), 0); + assert_eq!(isp.bar_read(1, isp_reg::MAILBOX_BASE, 2), 0); + assert_eq!(isp.bar_read(1, isp_reg::MAILBOX_BASE + 2, 2), 0x4953); +} + +/// Only admitted guest ports receive warm-reset notifications after both heads reset. +#[test] +fn warm_reset_notifies_only_admitted_ports_after_head_state_is_reset() { + let (c, _) = chip(); + let ports = Arc::new(Mutex::new(Vec::new())); + let output = ports.clone(); + let weak = Arc::downgrade(&c); + c.set_reset_output(Arc::new(move |port| { + if port >= 14 { + assert_eq!(weak.upgrade().unwrap().head(port).read32((reg::DIRECT_MAP + 4) as u64).data, 0x20000); + } + output.lock().push(port); + })); + let x = c.crossbar(); + x.write32((xbow::FENCE_BASE + 8 + 6 * 8 + 4) as u64, 0xff & !(1 << 2)); + c.head(14).write32((reg::DIRECT_MAP + 4) as u64, 0xa00000); + c.head(15).write32((reg::DIRECT_MAP + 4) as u64, 0xa00000); + c.warm_reset(10, 0); + assert_eq!(*ports.lock(), vec![8, 9, 10, 11, 12, 13, 15]); + assert_eq!(c.head(14).read32((reg::DIRECT_MAP + 4) as u64).data, 0xa00000); + ports.lock().clear(); + c.warm_reset(10, 10239); + assert!(ports.lock().is_empty()); + c.warm_reset(10, 10240); + assert_eq!(*ports.lock(), vec![8, 9, 10, 11, 12, 13, 14, 15]); +} +/// The crossbar reports the first error once and retains its programmed destination. +#[test] +fn crossbar_errors_deliver_once_then_clear_the_original_destination() { + let (c, _) = chip(); + let x = c.crossbar(); + let packets = Arc::new(Mutex::new(Vec::new())); + let out = packets.clone(); + c.set_interrupt_sender(Arc::new(move |p| { + out.lock().push(p); + true + })); + x.write32((xbow::INT_UPPER + 4) as u64, 0x550a0000); + x.write32((xbow::INT_LOWER + 4) as u64, 0x80); + x.write32((xbow::CONTROL + 4) as u64, 0x20); + x.read64(xbow::IDENTIFICATION as u64); + x.write32((xbow::IDENTIFICATION + 4) as u64, 1); + assert_eq!(packets.lock().len(), 1); + assert_eq!(x.read32((xbow::STATUS_CLEAR + 4) as u64).data & ((1 << 23) | 32), (1 << 23) | 32); + let p = packets.lock(); + assert_eq!(p.len(), 2); + assert_eq!(p[0].vector, 0x55); + assert_eq!(p[0].widget, 10); + assert_eq!(p[0].address, 0x80); + assert!(p[0].level); + assert!(!p[1].level); +} + +struct Card { + state: crate::pci::PciFunctionState, + bytes: Mutex>, +} +impl Card { + fn new() -> Arc { + Self::with_status_events(0) + } + fn with_status_events(mask: u16) -> Arc { + Self::with_bar(crate::pci::Bar::Mem(32), mask) + } + fn with_bar(bar: crate::pci::Bar, mask: u16) -> Arc { + use crate::pci::{Bar, PciConfig, PciFunctionState}; + let size = match bar { Bar::Mem(size) | Bar::Io(size) | Bar::Mem64(size) => size as usize, Bar::Unused => 0 }; + Arc::new(Self { + state: PciFunctionState::new(PciConfig::new_with_policy( + 0x1234, + 0x5678, + 0, + 0, + [ + bar, + Bar::Unused, + Bar::Unused, + Bar::Unused, + Bar::Unused, + Bar::Unused, + ], + 1, + crate::pci::PciConfigPolicy { status_w1c_mask: mask, ..Default::default() }, + )), + bytes: Mutex::new((0..size).map(|i| i as u8).collect()), + }) + } +} +impl PciFunction for Card { + fn state(&self) -> &crate::pci::PciFunctionState { + &self.state + } + fn bar_read(&self, _: usize, off: u32, width: usize) -> u32 { + let b = self.bytes.lock(); + (0..width).fold(0, |v, i| v | ((b[off as usize + i] as u32) << (i * 8))) + } + fn bar_write(&self, _: usize, off: u32, value: u32, width: usize) { + let mut b = self.bytes.lock(); + for i in 0..width { + b[off as usize + i] = (value >> (i * 8)) as u8; + } + } + fn reset(&self) { + if let Some(line) = self.state.irq() { + line.set(false); + } + } +} +/// The guest can identify the first invalid head access and its error destination. +#[test] +fn invalid_register_captures_first_address_and_reports_error_destination() { + let (c, _) = chip(); + let f = c.head(15); + let packets = Arc::new(Mutex::new(Vec::new())); + let out = packets.clone(); + c.set_interrupt_sender(Arc::new(move |p| { + out.lock().push(p); + true + })); + f.write32((reg::INT_DEST_UPPER + 4) as u64, 0xa0000); + f.write32((reg::INT_DEST_LOWER + 4) as u64, 0x80); + f.write32((reg::INTERRUPT_ERROR_HOST + 4) as u64, 73); + f.write32((reg::INTERRUPT_ENABLE + 4) as u64, 1 << 24); + f.read32(0x1000004); + f.read32(0x2000004); + assert_eq!(f.read32((reg::ERR_LOWER + 4) as u64).data, 0x1000004); + assert_eq!(packets.lock().len(), 1); + assert_eq!(packets.lock()[0].vector, 73); + f.write32((reg::INTERRUPT_RESET + 4) as u64, 8); + assert_eq!(f.read32((reg::INTERRUPT_STATUS + 4) as u64).data & (1 << 24), 0); +} +/// Narrow guest configuration writes preserve status bits outside the written bytes. +#[test] +fn partial_configuration_writes_preserve_unwritten_status_events() { + let (c, _) = chip(); + let f = c.head(15); + let card = Card::with_status_events(0xc000); + c.bus(15).attach(at(0), card.clone()); + for (bus, expected) in [(0, 0x56781234), (1, u32::MAX)] { + f.write32((reg::PCI_TYPE1_CONFIG + 4) as u64, bus << 16); + assert_eq!(f.read32(0x28000).data, expected); + } + card.state.config().set_status_bits(0xc000); + assert_eq!(f.write16(0x20006, 0x0102), BUS_OK); + assert_eq!(f.read32(0x20004).data, 0xc200_0102); + assert_eq!(f.write8(0x20007, 3), BUS_OK); + assert_eq!(f.read32(0x20004).data, 0xc200_0103); + assert_eq!(f.write8(0x20006, 2), BUS_OK); + assert_eq!(f.read32(0x20004).data, 0xc200_0203); + assert_eq!(f.write8(0x20005, 0xff), BUS_OK); + assert_eq!(f.read32(0x20004).data, 0xc200_0203); + assert_eq!(f.write8(0x20004, 0x80), BUS_OK); + assert_eq!(f.read32(0x20004).data, 0x4200_0203); + assert_eq!(f.write16(0x20004, 0x4000), BUS_OK); + assert_eq!(f.read32(0x20004).data, 0x0200_0203); + card.state.config().set_status_bits(0xc000); + assert_eq!(f.write32(0x20004, 0x8000_0002), BUS_OK); + assert_eq!(f.read32(0x20004).data, 0x4200_0002); +} + +fn assert_pio_matches_slow(head: &XBridgeHead, card: &Card, address: u64, mask: usize) { + for size in [1, 2, 4, 8] { + for offset in (0..32).step_by(size) { + let at = address + offset as u64; + let before = card.bytes.lock().clone(); + let expected = (0..size).fold(0u64, |v, i| (v << 8) | before[(offset + i) ^ mask] as u64); + assert_eq!(head.access(at, size, None), expected, "read {at:#x} width {size}"); + assert_eq!(head.slow_pio(at, size, None), expected); + head.access(at, size, Some(0x0123456789abcdef)); + let fast = card.bytes.lock().clone(); + *card.bytes.lock() = before.clone(); + head.slow_pio(at, size, Some(0x0123456789abcdef)); + assert_eq!(*card.bytes.lock(), fast, "write {at:#x} width {size}"); + *card.bytes.lock() = before; + } + } +} + +/// Native BAR dispatch preserves the bridge byte order at every supported access width. +#[test] +fn native_bar_pages_preserve_the_byte_order_at_every_width() { + for memory in [false, true] { + let (chip, _) = chip(); + let head = chip.pci_head(15); + let bar = if memory { crate::pci::Bar::Mem(0x2000) } else { crate::pci::Bar::Io(0x2000) }; + let card = Card::with_bar(bar, 0); + // Device window 0 reaches a function owned by slot 3. + chip.bus(15).attach(at(3), card.clone()); + head.write32(0x23010, 0x02000000 | u32::from(!memory)); + head.write32(0x23004, if memory { 2 } else { 1 }); + for wide in [false, true] { + chip.set_dma_64bit(15, 3, wide); + for stream in [false, true] { + head.write32((reg::DEVICE_BASE + 4) as u64, 0x20 | if memory { 1 << 12 } else { 0 } + | if wide { 1 << 22 } else { 0 } | if stream { 1 << 13 } else { 0 }); + assert!(runs(head).contains(&(0x200000, 0x202000))); + assert_pio_matches_slow(head, &card, 0x200000, if stream { 0 } else if wide { 7 } else { 3 }); + for swap in [false, true] { + head.write32((reg::CONTROL + 4) as u64, if swap { 1 << if memory { 22 } else { 23 } } else { 0 }); + let aliases = if memory { + vec![(0x42000000, false), (0x82000000, wide)] + } else { vec![(0x102000000, false)] }; + for (address, wide) in aliases { + assert_pio_matches_slow(head, &card, address, if swap { 0 } else if wide { 7 } else { 3 }); + } + } + } + } + } +} + +/// The runs the head serves without searching its bus. +fn runs(head: &XBridgeHead) -> Vec<(u64, u64)> { + head.pio.runs.read().iter().map(|run| (run.start, run.end)).collect() +} + +/// A moved, disabled or reset BAR stops reaching its former place. +#[test] +fn bar_moves_disable_and_reset_revoke_native_pages() { + let (chip, _) = chip(); + let head = chip.pci_head(15); + let card = Card::with_bar(crate::pci::Bar::Mem(0x1000), 0); + chip.bus(15).attach(at(0), card); + head.write32(0x20010, 0x02000000); + head.write32(0x20004, 2); + head.write32((reg::DEVICE_BASE + 4) as u64, 0x1020); + head.write32((reg::DEVICE_BASE + 4) as u64, 0x1022); + assert_eq!(head.read32(0x200000).data, u32::MAX); + head.write32((reg::DEVICE_BASE + 4) as u64, 0x1020); + assert_eq!(head.read32(0x200000).data, 0x03020100); + for i in 0..128 { + let base = 0x02000000 + i * 0x1000; + head.write32(0x20010, base); + assert_eq!(head.read32(0x40000000 + base as u64).data, 0x03020100); + if i != 0 { + assert_eq!(head.read32(0x40000000 + base as u64 - 0x1000).data, u32::MAX); + } + } + head.write8(0x20007, 0); + assert!(runs(head).is_empty()); + assert_eq!(head.read32(0x4207f000).data, u32::MAX); + head.write8(0x20007, 2); + assert!(!runs(head).is_empty()); + chip.crossbar().write32((xbow::LINK_BASE + 7 * xbow::LINK_STRIDE + xbow::LINK_RESET + 4) as u64, 1); + assert!(runs(head).is_empty()); + assert_eq!(head.read32(0x4207f000).data, u32::MAX); + head.write32(0x20010, 0x02000000); + head.write32(0x20004, 2); + assert_eq!(head.read32(0x42000000).data, 0x03020100); +} + +/// BARs that claim the same pages stay on the slow decoder, which takes the +/// first function in device order. +#[test] +fn overlapping_bar_pages_use_the_slow_decoder() { + let (chip, _) = chip(); + let head = chip.pci_head(15); + let small = Card::with_bar(crate::pci::Bar::Mem(0x1000), 0); + let large = Card::with_bar(crate::pci::Bar::Mem(0x2000), 0); + small.bytes.lock().fill(0xa5); + chip.bus(15).attach(at(0), small); + chip.bus(15).attach(at(1), large.clone()); + for slot in 0..2 { + head.write32(0x20010 + slot * 0x1000, 0x02000000); + head.write32(0x20004 + slot * 0x1000, 2); + } + head.write32((reg::DEVICE_BASE + 4) as u64, 0x1020); + assert!(runs(head).is_empty()); + assert_eq!(head.read32(0x42000000).data, 0xa5a5a5a5); + assert_eq!(head.read32(0x42001000).data, 0x03020100, "past the small BAR, only the large one decodes"); + head.write32(0x20004, 0); + assert!(!runs(head).is_empty()); + assert_pio_matches_slow(head, &large, 0x42000000, 3); + head.write32(0x21004, 0); + head.write32(0x20004, 2); + assert!(!runs(head).is_empty()); + assert_eq!(head.read32(0x42000000).data, 0xa5a5a5a5); +} + +/// BAR holes and cross-boundary cycles must use the original slow decoder, +/// including after the host publishes a replacement set of BAR mappings. +#[test] +fn sparse_bar_lookup_rejects_holes_and_cross_boundary_cycles() { + let decode = PioDecode::default(); + let first: Arc = Arc::new(Memory(Mutex::new(vec![0; 0x4000]))); + let second: Arc = Arc::new(Memory(Mutex::new(vec![0; 0x4000]))); + drop(decode.publish(vec![ + PioRun { start: 0x3000, end: 0x4000, dev: second.clone() }, + PioRun { start: 0x1000, end: 0x2000, dev: first.clone() }, + ])); + assert!(Arc::ptr_eq(&decode.target(0x1000, 8).unwrap(), &first)); + assert!(Arc::ptr_eq(&decode.target(0x1ff8, 8).unwrap(), &first)); + assert!(decode.target(0x1ffc, 8).is_none()); + assert!(decode.target(0x2000, 8).is_none()); + assert!(decode.target(0x2ff8, 8).is_none()); + assert!(Arc::ptr_eq(&decode.target(0x3000, 8).unwrap(), &second)); + assert!(decode.target(0x4000, 8).is_none()); + assert!(decode.target(u64::MAX, 8).is_none()); + drop(decode.publish(vec![PioRun { start: 0x2000, end: 0x3000, dev: first.clone() }])); + assert!(decode.target(0x1000, 8).is_none()); + assert!(Arc::ptr_eq(&decode.target(0x2000, 8).unwrap(), &first)); + assert!(decode.target(0x3000, 8).is_none()); +} + +fn xbow_read(x: &Arc, off: u32) -> u32 { + x.read32((off + 4) as u64).data +} + +/// Software probing the crossbar after reset sees these values. Each +/// link's control word resets to 0x80000; that value belongs in the control +/// word, not in the link reset register, which no read can see. +#[test] +fn the_crossbar_resets_to_its_documented_values() { + let (c, _) = chip(); + let x = c.crossbar(); + for (off, value) in [ + (xbow::ERR_UPPER, 0), + (xbow::ERR_LOWER, 0), + (xbow::CONTROL, 0x80), + (xbow::PACKET_TIMEOUT, 0xfffff), + (xbow::INT_UPPER, 0x80000), + (xbow::INT_LOWER, 0), + (xbow::ERR_CMD, 0), + (xbow::LLP_CONTROL, 0x03ff1810), + (xbow::ARB_RELOAD, 2), + (xbow::PERF_A, 0), + (xbow::PERF_B, 0), + (xbow::NIC, 2), + (xbow::FENCE_BASE, 0xff), + (xbow::LOCK, 0), + ] { + assert_eq!(xbow_read(&x, off), value, "{off:#x}"); + } + for link in 0..8 { + let base = xbow::LINK_BASE + link * xbow::LINK_STRIDE; + assert_eq!(xbow_read(&x, xbow::FENCE_BASE + 8 + link * 8), 0xff); + assert_eq!(xbow_read(&x, base + xbow::LINK_CONTROL), 0x80000, "link {}", link + 8); + assert_eq!(xbow_read(&x, base + xbow::LINK_ARB_UPPER), 0); + assert_eq!(xbow_read(&x, base + xbow::LINK_ARB_LOWER), 0); + } +} + +/// Software shares the crossbar between processors with the resource lock: +/// a write takes it only while it is free, so a second holder is refused +/// until the first writes the clear register. +#[test] +fn the_resource_lock_admits_one_holder_until_cleared() { + let (c, _) = chip(); + let x = c.crossbar(); + let other = c.crossbar_for_port(HEAD_E); + x.write32((xbow::LOCK + 4) as u64, 0x5a); + assert_eq!(xbow_read(&x, xbow::LOCK), 0x5a); + other.write32((xbow::LOCK + 4) as u64, 0x3c); + assert_eq!(xbow_read(&x, xbow::LOCK), 0x5a); + assert_eq!(xbow_read(&x, xbow::LOCK_CLEAR), 0x5a); + x.write32((xbow::LOCK_CLEAR + 4) as u64, 0); + assert_eq!(xbow_read(&x, xbow::LOCK), 0); + other.write32((xbow::LOCK + 4) as u64, 0x3c); + assert_eq!(xbow_read(&x, xbow::LOCK), 0x3c); +} + +/// Every reset fence bit is read and write, a link's own bit included, and +/// only the low eight bits exist. +#[test] +fn reset_fences_keep_their_low_eight_bits() { + let (c, _) = chip(); + let x = c.crossbar(); + for off in (xbow::FENCE_BASE..=FENCE_LAST).step_by(8) { + x.write32((off + 4) as u64, 0); + assert_eq!(xbow_read(&x, off), 0); + x.write32((off + 4) as u64, off as u32); + assert_eq!(xbow_read(&x, off), off as u32 & 0xff); + x.write32((off + 4) as u64, u32::MAX); + assert_eq!(xbow_read(&x, off), 0xff); + } +} diff --git a/src/devlog.rs b/src/devlog.rs index 7c481b3c5..7e0506404 100644 --- a/src/devlog.rs +++ b/src/devlog.rs @@ -58,6 +58,17 @@ pub enum LogModule { Bedrock = 29, /// The SN0 node controller: guest accesses the model does not support. Hub = 30, + /// PCI configuration cycles on every bus: who was addressed, what was + /// read or written, and whether anything answered. + Pci = 31, + /// The Octane's Bridge ASIC: register and translation-table accesses, + /// and transfers a function started that reached no memory. + Bridge = 32, + /// The Octane's HEART: processor-interface register accesses (not + /// counter reads) and interrupt vectors raised and lowered. + Heart = 33, + L1 = 34, + Usb = 35, } // Mask bits shared by L1i / L1d / L2c @@ -66,7 +77,7 @@ pub const CACHE_LOG_MISS: u32 = 0x2; // log misses and fills pub const CACHE_LOG_OP: u32 = 0x4; // log CACHE instruction ops impl LogModule { - pub const COUNT: usize = 31; + pub const COUNT: usize = 36; pub fn from_str(s: &str) -> Option { match s { @@ -101,6 +112,11 @@ impl LogModule { "odyssey" => Some(Self::Odyssey), "bedrock" => Some(Self::Bedrock), "hub" => Some(Self::Hub), + "pci" => Some(Self::Pci), + "bridge" => Some(Self::Bridge), + "heart" => Some(Self::Heart), + "l1" => Some(Self::L1), + "usb" => Some(Self::Usb), _ => None, } } @@ -138,6 +154,11 @@ impl LogModule { Self::Odyssey => "odyssey", Self::Bedrock => "bedrock", Self::Hub => "hub", + Self::Pci => "pci", + Self::Bridge => "bridge", + Self::Heart => "heart", + Self::L1 => "l1", + Self::Usb => "usb", } } @@ -149,7 +170,8 @@ impl LogModule { Self::Bt445, Self::Scc, Self::Ps2, Self::Rtc, Self::Eeprom, Self::L1i, Self::L1d, Self::L2c, Self::Ultra, Self::Scsi1, Self::Nveeprom, Self::Gr2, Self::Rad1, Self::Odyssey, Self::Bedrock, - Self::Hub, + Self::Hub, Self::Pci, Self::Bridge, Self::Heart, + Self::L1, Self::Usb, ] } @@ -293,6 +315,11 @@ impl DevLog { ModuleLog::new(), // Odyssey ModuleLog::new(), // Bedrock ModuleLog::new(), // Hub + ModuleLog::new(), // Pci + ModuleLog::new(), // Bridge + ModuleLog::new(), // Heart + ModuleLog::new(), // L1 + ModuleLog::new(), // Usb ], } } @@ -480,7 +507,7 @@ impl Device for DevLog { vec![( "log".to_string(), "[DEV] log | log mask | log file | log status\n\ - \x20 modules: net hpc3 seeq hal2 mc rex3 mips ioc scsi pdma vino dcb vc2 cmap xmap bt445 scc ps2 rtc eeprom l1i l1d l2c\n\ + \x20 modules: net hpc3 seeq hal2 mc rex3 mips ioc scsi pdma vino dcb vc2 cmap xmap bt445 scc ps2 rtc eeprom l1i l1d l2c pci bridge heart\n\ \x20 pdma mask categories: hal enet scsi on/all off/none \n\ \x20 mips mask categories: insn tlb mem fpu cp0 on/all off/none \n\ \x20 l1i/l1d/l2c mask categories: hit miss op on/all off/none \n\ diff --git a/src/exit_persistence_tests.rs b/src/exit_persistence_tests.rs index 622e229ca..896991863 100644 --- a/src/exit_persistence_tests.rs +++ b/src/exit_persistence_tests.rs @@ -410,7 +410,7 @@ fn wait_for_exit_request(before: usize) { /// The guest's own power-off, down the device path each machine has. fn guest_power_off(m: &mut dyn Workstation) { use crate::dev::ioc3::reg; - use crate::pci::PciBusDevice; + use crate::pci::PciFunction; use crate::traits::BusDevice; let any = m.as_any_mut(); if let Some(m) = any.downcast_mut::() { diff --git a/src/ip27_machine.rs b/src/ip27_machine.rs index 73bfd0a98..b8bce13d8 100644 --- a/src/ip27_machine.rs +++ b/src/ip27_machine.rs @@ -497,21 +497,21 @@ impl Ip27Machine { } else { crate::with_cpu!(module.part, true, false, C => Self::cpu::(module, bus, phys, memory, &node.timers, count_hz, cpu_id, smp)) }; - // The shadow cache and the JIT read RAM straight from the ppmem window. + // The shadow cache and the JIT read RAM through a cached view of the + // ppmem window that is the CPU's own. if let Some(sp) = node.memory.physical().ppmem_space() { - use crate::ppmem::MappedMemory; + let v = sp + .add_cached_view(&[cpu.ppmem_bitmap_ptr(), cpu.tcache_bitmap_ptr()]) + .map_err(|e| format!("ppmem: a CPU's cached view: {e}"))?; unsafe { - sp.set_bitmap_sink(cpu.ppmem_bitmap_ptr()); - cpu.set_tcache_window(sp.window_base()); + cpu.set_tcache_window(sp.view_base(v)); cpu.set_tcache_shift(crate::ppmem::window_shift(sp.window_size())); - let bitmap = cpu.tcache_bitmap_ptr(); - if !bitmap.is_null() { - sp.set_bitmap_sink2(bitmap); - } #[cfg(feature = "jitv2")] - cpu.set_tcache_gen_window(sp.gen_window_base()); + { + use crate::ppmem::MappedMemory; + cpu.set_tcache_gen_window(sp.gen_window_base()); + } } - node.memory.physical().resync_ppmem_bitmap(); } #[cfg(feature = "jitv2")] { @@ -848,7 +848,7 @@ impl Ip27Board for Origin200 { fn wire(nodes: &[&Node], cfg: &MachineConfig) -> Result { let node = nodes[0]; let controller = Origin200Controller::new(node, cfg.origin200.controller_store.as_deref().map(Path::new))?; - let bridge = Bridge::new(Upstream::single(HUB_HOST, node.hub.inbound(), node.hub.interrupt_target())); + let bridge = Bridge::new(Upstream::single(HUB_HOST, node.hub.inbound(), node.hub.interrupt_target()), &crate::ip30::PCI0); bridge.set_revision(BRIDGE_REVISION); bridge.set_microlan(Self::board_record()); // The Bridge answers every small window but the Hub's own. @@ -887,14 +887,16 @@ pub(crate) fn pci_devices( for s in &scsi { s.set_heartbeat(heartbeat.clone()); } - bridge.attach(0, scsi[0].clone()); - bridge.attach(1, scsi[1].clone()); + let pci = bridge.bus(); + let slot = |device| crate::pci::DevFn { device, function: 0 }; + pci.attach(slot(0), scsi[0].clone()); + pci.attach(slot(1), scsi[1].clone()); let ioc3 = Arc::new(Ioc3::new(ioc3_config)); ioc3.set_heartbeat(heartbeat.clone()); eth.set_heartbeat(heartbeat); ioc3.attach_eth(eth.clone()); - bridge.attach(2, ioc3.clone()); - ioc3.connect_superio_interrupt(bridge.pin_line(superio_pin)); + pci.attach(slot(2), ioc3.clone()); + ioc3.connect_superio_interrupt(pci.board_irq(crate::pci::BoardIrq(superio_pin))); ioc3.set_timer_manager(node.timers.clone()); // The Fuel's M48T35 Timekeeper is the same part. let timekeeper = Arc::new(Timekeeper::fuel_with_nvram_file(Path::new(&cfg.nvram))?); diff --git a/src/ip27_onyx2.rs b/src/ip27_onyx2.rs index 91dbd48dc..d981eff52 100644 --- a/src/ip27_onyx2.rs +++ b/src/ip27_onyx2.rs @@ -153,7 +153,7 @@ impl Ip27Board for Onyx2Deskside { .enumerate() .map(|(i, n)| HostPort::new(hub_host(i), n.hub.inbound(), n.hub.interrupt_target())) .collect(); - let bridge = Bridge::new(xbow.upstream(hosts)); + let bridge = Bridge::new(xbow.upstream(hosts), &crate::ip30::PCI0); bridge.set_revision(BRIDGE_REVISION); bridge.set_microlan(Self::board_record()); let links = xbow.clone(); diff --git a/src/ip27_onyx2/tests.rs b/src/ip27_onyx2/tests.rs index e13a7daa5..e9fa79042 100644 --- a/src/ip27_onyx2/tests.rs +++ b/src/ip27_onyx2/tests.rs @@ -146,35 +146,32 @@ fn a_dropped_machine_frees_its_hubs_memory_and_timers() { /// A PCI function with nothing behind it but the DMA and interrupt line the /// Bridge hands it. struct Card { - cfg: crate::pci::PciConfig, - irq: parking_lot::Mutex>, - dma: parking_lot::Mutex>>, + state: crate::pci::PciFunctionState, } impl Card { fn new() -> Arc { Arc::new(Self { - cfg: crate::pci::PciConfig::new(0x1234, 0x5678, 0x01_0000, 1, [crate::pci::Bar::Unused; 6], crate::pci::PCI_INTA), - irq: Default::default(), - dma: Default::default(), + state: crate::pci::PciFunctionState::new(crate::pci::PciConfig::new( + 0x1234, + 0x5678, + 0x01_0000, + 1, + [crate::pci::Bar::Unused; 6], + crate::pci::PCI_INTA, + )), }) } } -impl crate::pci::PciBusDevice for Card { - fn config(&self) -> &crate::pci::PciConfig { - &self.cfg +impl crate::pci::PciFunction for Card { + fn state(&self) -> &crate::pci::PciFunctionState { + &self.state } fn bar_read(&self, _bar: usize, _off: u32, _width: usize) -> u32 { 0 } fn bar_write(&self, _bar: usize, _off: u32, _val: u32, _width: usize) {} - fn connect_interrupt(&self, line: crate::irq::IrqLine) { - *self.irq.lock() = Some(line); - } - fn connect_dma(&self, dma: Arc) { - *self.dma.lock() = Some(dma); - } } /// The BaseIO-G's Bridge sends DMA and interrupts to the Hub at the widget @@ -188,9 +185,9 @@ fn the_bridge_sends_dma_and_interrupts_to_the_hub_the_guest_names() { let (_dir, m) = built("bridge", 2); let bridge = m.bridge().clone(); let card = Card::new(); - bridge.attach(7, card.clone()); + bridge.bus().attach(crate::pci::DevFn { device: 7, function: 0 }, card.clone()); let set = |off: u32, v: u32| assert_eq!(bridge.write32(u64::from(off + 4), v), BUS_OK); - let dma = card.dma.lock().clone().unwrap(); + let dma = card.state.dma().unwrap().clone(); let memory = |i: usize, off: u64| m.node_at(i).memory.physical().read32(off).data; let pending = |i: usize| m.node_at(i).hub.read64(crate::dev::hub_pi::reg::INT_PEND0).data; @@ -207,10 +204,10 @@ fn the_bridge_sends_dma_and_interrupts_to_the_hub_the_guest_names() { set(reg::INT_DEST_LOWER, 0); set(reg::int_host(7), 5); set(reg::INT_ENABLE, 1 << 7); - card.irq.lock().as_ref().unwrap().set(true); + card.state.irq().unwrap().set(true); assert_eq!(pending(node) & 1 << 5, 1 << 5, "widget {widget}"); assert_eq!(pending(other) & 1 << 5, 0, "widget {widget}: the other node heard it"); - card.irq.lock().as_ref().unwrap().set(false); + card.state.irq().unwrap().set(false); // The guest's handler clears the pending bit. assert_eq!(m.node_at(node).hub.write64(crate::dev::hub_pi::reg::INT_PEND_MOD, 5), BUS_OK); } diff --git a/src/ip27_origin2000.rs b/src/ip27_origin2000.rs index c9e210e0a..0344816db 100644 --- a/src/ip27_origin2000.rs +++ b/src/ip27_origin2000.rs @@ -217,7 +217,7 @@ impl Ip27Board for Origin2000 { node.hub.io().connect_widget(host.widget, xbow.port(host.widget, node.hub.widget_face())); xbow.register_locks(); if i == 0 { - let bridge = Bridge::new(Upstream::single(host, node.hub.inbound(), node.hub.interrupt_target())); + let bridge = Bridge::new(Upstream::single(host, node.hub.inbound(), node.hub.interrupt_target()), &crate::ip30::PCI0); bridge.set_revision(BRIDGE_REVISION); bridge.set_microlan(Self::board_record()); let links = xbow.clone(); diff --git a/src/ip30.rs b/src/ip30.rs index 4bc57da89..260cbfe74 100644 --- a/src/ip30.rs +++ b/src/ip30.rs @@ -11,6 +11,7 @@ //! | `0x0ff0_0000` | HEART's processor-interface registers | //! | `0x1000_0000 + (w << 24)` | widget `w`'s 16 MiB small window | //! | `0x2000_0000` up | SDRAM through HEART's bank descriptors (`heart_mem::HeartRam`); its first 16 KiB bus-error here | +//! | `0x8_0000_0000 + w × 2 GiB` | widget `w`'s 2 GiB window (the Bridge's only) | //! | `w << 36` | widget `w`'s 64 GiB large window (8 to 15) | //! //! Widget 0 is the Crossbow, 8 HEART, 15 the onboard Bridge (whose window @@ -24,18 +25,41 @@ //! //! Set IRIS_IP30_TRACE=1 to log unmapped accesses as they happen. +use std::ops::Range; use std::sync::atomic::{AtomicBool, AtomicU64, Ordering}; use std::sync::Arc; use parking_lot::Mutex; use crate::dev::heart_mem::HeartRam; -use crate::pci::BarRun; +use crate::pci::IntxRouting; use crate::physical::Physical; use crate::ppmem::devmap::{entry_eq, SLOT_SHIFT}; use crate::traits::{BusDevice, BusRead16, BusRead32, BusRead64, BusRead8, BUS_ERR, BUS_OK}; /// Widget numbers on the Octane's Crossbow. +/// The Bridge's PCI bus; the outputs are the Bridge's eight interrupt pins. +/// Pins A and C of a device reach the pin numbered as the device, B and D +/// that number with bit two flipped. That rule is inferred from the system's +/// software and is not established by a board document. `BoardIrq(i)` is the +/// Bridge's pin `i`, which the GPIO inputs share with PCI; the IOC3's +/// SuperIO interrupt is `BoardIrq(4)`. The wire numbering is iris's own. +pub static PCI0: IntxRouting = IntxRouting { + devices: 8, + irqs: 8, + intx: &[ + (0, [Some(0), Some(4), Some(0), Some(4)]), + (1, [Some(1), Some(5), Some(1), Some(5)]), + (2, [Some(2), Some(6), Some(2), Some(6)]), + (3, [Some(3), Some(7), Some(3), Some(7)]), + (4, [Some(4), Some(0), Some(4), Some(0)]), + (5, [Some(5), Some(1), Some(5), Some(1)]), + (6, [Some(6), Some(2), Some(6), Some(2)]), + (7, [Some(7), Some(3), Some(7), Some(3)]), + ], + board_irqs: &[0, 1, 2, 3, 4, 5, 6, 7], +}; + pub const WIDGET_XBOW: u8 = 0x0; pub const WIDGET_HEART: u8 = 0x8; pub const WIDGET_BRIDGE: u8 = 0xf; @@ -55,6 +79,12 @@ pub const fn widget_large_base(w: u8) -> u64 { (w as u64) << 36 } pub const WIDGET_LARGE_SIZE: u64 = 1 << 36; +/// Widget `w`'s 2 GiB window, the first 2 GiB of its XIO space, at +/// `0x8_0000_0000 + w × 2 GiB` (`docs/octane-ip30-address-map.md`). +pub const fn widget_long_base(w: u8) -> u64 { + 0x8_0000_0000 + (w as u64) * WIDGET_LONG_SIZE +} +pub const WIDGET_LONG_SIZE: u64 = 0x8000_0000; /// Where SDRAM space ends: the first large window. pub const SDRAM_SPACE_END: u64 = crate::dev::heart_mem::SDRAM_END; const _: () = assert!(SDRAM_SPACE_END == widget_large_base(8)); @@ -141,10 +171,13 @@ pub struct Ip30Bus { /// The widgets on the Crossbow's ports 9 to 15, and whether each also /// takes its large window. widgets: Mutex<[Option<(Arc, bool)>; 16]>, - /// Per port, the runs of its small window that a PCI BAR decodes whole - /// (the Bridge's [`crate::pci::BarRun`]s, offsets in the window), mapped - /// over the widget while its link is up. - runs: Mutex<[Vec; 16]>, + /// Per port, what answers its 2 GiB window, if anything does. + long: Mutex<[Option>; 16]>, + /// Per port, the runs that a PCI BAR decodes whole (the Bridge's runs), + /// mapped over the widget while its link is up. A run starting below + /// 16 MiB is in the small window, any other in the 2 GiB window; both + /// are offsets in XIO space. + runs: Mutex<[Vec<(Range, Arc)>; 16]>, } impl Ip30Bus { @@ -156,7 +189,14 @@ impl Ip30Bus { let unmapped = Arc::new(Unmapped::default()); phys.set_default(unmapped.clone(), "unmapped (bus error)"); ram.rebuild(); - Self { phys, ram, unmapped, widgets: Mutex::new(Default::default()), runs: Mutex::new(Default::default()) } + Self { + phys, + ram, + unmapped, + widgets: Mutex::new(Default::default()), + long: Mutex::new(Default::default()), + runs: Mutex::new(Default::default()), + } } /// Put `dev` at physical `start..end` (`end` exclusive, rounded up to a @@ -175,6 +215,13 @@ impl Ip30Bus { self.widgets.lock()[w as usize] = Some((dev, large)); } + /// Put `dev` behind port `w`'s 2 GiB window, gated by the link as the + /// small window is. Call before [`Self::init`]. + pub fn attach_long_window(&self, w: u8, dev: Arc) { + assert!((9..16).contains(&w), "ip30: widget {w} is not a port the Crossbow switches"); + self.long.lock()[w as usize] = Some(dev); + } + /// Fill the widget windows as each link stands (dead). Call once, after /// the devices are registered and before the CPU runs. pub fn init(&self) { @@ -203,13 +250,16 @@ impl Ip30Bus { self.unmap(widget_small_base(w), WIDGET_SMALL_SIZE); return; }; - let mut windows = vec![(widget_small_base(w), WIDGET_SMALL_SIZE)]; + let mut windows = vec![(widget_small_base(w), WIDGET_SMALL_SIZE, dev.clone())]; if *large { - windows.push((widget_large_base(w), WIDGET_LARGE_SIZE)); + windows.push((widget_large_base(w), WIDGET_LARGE_SIZE, dev.clone())); + } + if let Some(long) = &self.long.lock()[w as usize] { + windows.push((widget_long_base(w), WIDGET_LONG_SIZE, long.clone())); } - for (base, size) in windows { + for (base, size, dev) in windows { if alive { - self.phys.map(base, base + size, dev.clone(), &format!("widget {w:#x}")); + self.phys.map(base, base + size, dev, &format!("widget {w:#x}")); } else { self.unmap(base, size); } @@ -220,9 +270,9 @@ impl Ip30Bus { } /// Widget `w` says which runs of its small window a PCI BAR decodes - /// whole (the Bridge's `on_bar_runs`): map them over the widget while + /// whole (the Bridge's BAR runs): map them over the widget while /// its link is up, giving the runs it named before back to the widget. - pub fn set_widget_runs(&self, w: u8, new: Vec) { + pub fn set_widget_runs(&self, w: u8, new: Vec<(Range, Arc)>) { if !(9..16).contains(&w) { return; } @@ -230,22 +280,40 @@ impl Ip30Bus { let mut runs = self.runs.lock(); let old = std::mem::replace(&mut runs[w as usize], new); let Some((dev, _)) = &widgets[w as usize] else { return }; - let base = widget_small_base(w); // The window's first page is the widget's own registers, never a // run: it answers as the widget exactly while the link is up. - if !entry_eq(self.phys.entry_at(base), Arc::as_ptr(dev)) { + if !entry_eq(self.phys.entry_at(widget_small_base(w)), Arc::as_ptr(dev)) { return; // link down: `set_widget_link` maps them when it comes up } - for r in &old { - self.phys.map(base + r.start, base + r.end, dev.clone(), &format!("widget {w:#x}")); + let long = self.long.lock()[w as usize].clone(); + for (r, _) in &old { + let Some((base, under)) = Self::run_window(w, r, dev, long.as_ref()) else { continue }; + self.phys.map(base + r.start, base + r.end, under, &format!("widget {w:#x}")); } self.map_runs(w, &runs[w as usize]); } - fn map_runs(&self, w: u8, runs: &[BarRun]) { - let base = widget_small_base(w); - for r in runs { - self.phys.map(base + r.start, base + r.end, r.dev.clone(), &format!("widget {w:#x} pci bar")); + /// Where run `r` of widget `w` lies and what answers there without it: a + /// run below 16 MiB is in the small window (the widget), any other in the + /// 2 GiB window (`long`). None for a run in a 2 GiB window the widget does + /// not have, which is not mapped at all. + fn run_window( + w: u8, + r: &Range, + dev: &Arc, + long: Option<&Arc>, + ) -> Option<(u64, Arc)> { + if r.start < WIDGET_SMALL_SIZE { + return Some((widget_small_base(w), dev.clone())); + } + long.map(|l| (widget_long_base(w), l.clone())) + } + + fn map_runs(&self, w: u8, runs: &[(Range, Arc)]) { + let long = self.long.lock()[w as usize].clone(); + for (r, dev) in runs { + let Some((base, _)) = Self::run_window(w, r, dev, long.as_ref()) else { continue }; + self.phys.map(base + r.start, base + r.end, dev.clone(), &format!("widget {w:#x} pci bar")); } } diff --git a/src/ip30/tests.rs b/src/ip30/tests.rs index 9b4ee769e..6aa4b2f07 100644 --- a/src/ip30/tests.rs +++ b/src/ip30/tests.rs @@ -64,13 +64,14 @@ fn board_with(prom: &[u8], banks_mb: [u32; 4]) -> Board { let heart = Arc::new(Heart::new(HeartConfig::octane(banks_mb.map(|mb| (mb as u64) << 20)), crate::dev::heart::host_clock())); let xbow = Arc::new(Xbow::new(XBOW_REVISION, 1 << WIDGET_HEART | 1 << WIDGET_BRIDGE)); xbow.connect_host(HEART_HOST, heart.clone()); - let bridge = Bridge::new(Upstream::single(HEART_HOST, ram.dma_view(), heart.clone())); + let bridge = Bridge::new(Upstream::single(HEART_HOST, ram.dma_view(), heart.clone()), &crate::ip30::PCI0); bridge.attach_flash(Arc::new(RawFlash::new(prom, FLASH_SIZE)), FLASH_SIZE as u32); let phys = Ip30Bus::new(ram); phys.map(PIU_BASE, PIU_BASE + PIU_SIZE, heart.clone(), "heart piu"); phys.map(widget_small_base(WIDGET_XBOW), widget_small_base(WIDGET_XBOW) + WIDGET_SMALL_SIZE, xbow.clone(), "xbow"); phys.map(widget_small_base(WIDGET_HEART), widget_small_base(WIDGET_HEART) + WIDGET_SMALL_SIZE, heart.clone(), "heart widget"); - phys.attach_widget(WIDGET_BRIDGE, bridge, false); + phys.attach_widget(WIDGET_BRIDGE, bridge.clone(), false); + phys.attach_long_window(WIDGET_BRIDGE, bridge.long_window()); let phys = Arc::new(phys); phys.init(); for w in 9..16 { @@ -390,6 +391,60 @@ fn widgets_route_by_number() { assert_eq!(p.read32(widget_large_base(WIDGET_BRIDGE)).status, BUS_ERR, "the Bridge takes no large window"); } +/// The Bridge's 2 GiB window at 0xF_8000_0000 starts with its small +/// window, and a BAR run in its PCI memory alias is mapped there, not in +/// the small window. IRIX 6.5.30 reads a BAR behind a PCI-to-PCI bridge at +/// 0xF_C080_0008 (PCI memory 0x80_0008). +#[test] +fn the_bridge_2gb_window_holds_its_registers_and_alias_runs() { + let b = board(&[]); + let p = &b.phys; + let long = widget_long_base(WIDGET_BRIDGE); + assert_eq!(long, 0xf_8000_0000); + assert_eq!(p.read32(long + 4).data, p.read32(widget_small_base(WIDGET_BRIDGE) + 4).data, "identification"); + assert_eq!(p.read32(long + 0x0100_0000).status, BUS_ERR, "nothing past the small window's 16 MiB"); + let at = PIO32_ALIAS as u64 + 0x80_0000; + let marker: Arc = Arc::new(Marker(0x5a5a_5a5a)); + b.bus.set_widget_runs(WIDGET_BRIDGE, vec![(at..at + 0x2000, marker)]); + assert_eq!(p.read32(long + at + 8).data, 0x5a5a_5a5a, "the run answers in the 2 GiB window"); + assert_eq!(p.read32(widget_small_base(WIDGET_BRIDGE) + 0x80_0008).status, BUS_ERR, "and not in the small window"); + b.bus.set_widget_runs(WIDGET_BRIDGE, vec![]); + assert_eq!(p.read32(long + at + 8).status, BUS_ERR, "given back: a master time-out, with nothing decoding"); +} + +/// A widget with no 2 GiB window attached keeps a run above 16 MiB out of +/// the bus altogether: it is not mapped, and dropping it maps nothing back +/// (the small window's base plus the run's offset is SDRAM space). +#[test] +fn a_run_above_16_mib_without_a_2gb_window_maps_nothing() { + let ram = Arc::new(HeartRam::new([128, 0, 0, 0])); + let heart = Arc::new(Heart::new(HeartConfig::octane([128 << 20, 0, 0, 0]), crate::dev::heart::host_clock())); + let bridge = Bridge::new(Upstream::single(HEART_HOST, ram.dma_view(), heart), &PCI0); + let p = Arc::new(Ip30Bus::new(ram)); + let dev: Arc = bridge; + p.attach_widget(WIDGET_BRIDGE, dev, false); + p.init(); + p.set_widget_link(WIDGET_BRIDGE, true); + let at = PIO32_ALIAS as u64 + 0x80_0000; + let marker: Arc = Arc::new(Marker(0x5a5a_5a5a)); + let (small, long) = (widget_small_base(WIDGET_BRIDGE) + at, widget_long_base(WIDGET_BRIDGE) + at); + let empty = p.phys.default_entry(); + p.set_widget_runs(WIDGET_BRIDGE, vec![(at..at + 0x2000, marker)]); + assert!(entry_eq(p.phys.entry_at(long), empty), "no run in a window it does not have"); + p.set_widget_runs(WIDGET_BRIDGE, vec![]); + assert!(entry_eq(p.phys.entry_at(small), empty), "nothing given back over memory space"); + assert!(entry_eq(p.phys.entry_at(long), empty)); +} + +/// A device that answers every word read with one value. +struct Marker(u32); + +impl BusDevice for Marker { + fn read32(&self, _: u64) -> crate::traits::BusRead32 { + crate::traits::BusRead32::ok(self.0) + } +} + #[test] fn the_graphics_port_reaches_impact_with_its_link_up() { let ram = Arc::new(HeartRam::new([128, 0, 0, 0])); @@ -463,19 +518,20 @@ impl Octane { ram.set_memcfg([0x8001_0000, 0, 0, 0]); let mem = ram.dma_view(); let heart = Arc::new(Heart::new(HeartConfig::octane([64 << 20, 0, 0, 0]), Arc::new(|| 0))); - let bridge = Bridge::new(Upstream::single(HEART_HOST, mem.clone(), heart.clone())); + let bridge = Bridge::new(Upstream::single(HEART_HOST, mem.clone(), heart.clone()), &crate::ip30::PCI0); let eth = Ioc3Eth::with_negotiation_delay(MAC, std::time::Duration::ZERO); let (link, sent, inject) = crate::net::Link::loopback(); eth.attach(link); let ioc3 = Arc::new(Ioc3::octane(MAC, false)); ioc3.attach_eth(eth.clone()); - ioc3.connect_superio_interrupt(bridge.pin_line(SUPERIO_PIN as u8)); + ioc3.connect_superio_interrupt(bridge.bus().board_irq(crate::pci::BoardIrq(SUPERIO_PIN as u8))); let isp = [Arc::new(Isp1040::new_at(0, Isp1040::new_bus())), Arc::new(Isp1040::new_at(1, Isp1040::new_bus()))]; - bridge.attach(SLOT_ISP0 as u8, isp[0].clone()); - bridge.attach(SLOT_ISP1 as u8, isp[1].clone()); - bridge.attach(SLOT_IOC3 as u8, ioc3.clone()); + let at = |slot: u32| crate::pci::DevFn { device: slot as u8, function: 0 }; + bridge.bus().attach(at(SLOT_ISP0 as u32), isp[0].clone()); + bridge.bus().attach(at(SLOT_ISP1 as u32), isp[1].clone()); + bridge.bus().attach(at(SLOT_IOC3 as u32), ioc3.clone()); Octane { heart, mem, bridge, isp, sent, inject, eth } } @@ -1066,26 +1122,29 @@ fn flash_serves_the_stock_prom_as_the_processor_sees_it() { #[test] fn a_bar_run_answers_as_the_window_does() { let o = Octane::new(); - let runs = Arc::new(parking_lot::Mutex::new(Vec::::new())); - let sink = runs.clone(); - o.bridge.on_bar_runs(Box::new(move |r| *sink.lock() = r)); + let runs = Arc::new(parking_lot::Mutex::new(Vec::<(std::ops::Range, Arc)>::new())); + let (sink, bridge) = (runs.clone(), Arc::downgrade(&o.bridge)); + o.bridge.bus().on_bar_mappings(Box::new(move |r| { + let b = bridge.upgrade().unwrap(); + *sink.lock() = r.iter().map(|r| (r.cpu..r.cpu + r.len, b.bar_device(r))).collect(); + })); assert!(runs.lock().is_empty(), "nothing decodes before the PROM places a BAR"); o.ioc3_map(); let win = Octane::ioc3_win() as u64; - let run = runs.lock().iter().find(|r| r.start <= win && win < r.end).cloned().expect("IOC3's window is a run"); - assert_eq!((run.start, run.end), (win, win + (1 << 20)), "the whole 1 MB window"); + let run = runs.lock().iter().find(|r| r.0.contains(&win)).cloned().expect("IOC3's window is a run"); + assert_eq!((run.0.start, run.0.end), (win, win + (1 << 20)), "the whole 1 MB window"); let at = win + RTC_DATA as u64; o.w8(RTC_INDEX, 0x0e); // a byte of the clock's RAM - assert_eq!(run.dev.write32(at, 0xaabb_ccdd), BUS_OK); + assert_eq!(run.1.write32(at, 0xaabb_ccdd), BUS_OK); assert_eq!(o.i8(RTC_DATA), 0xaa, "a word store through the run lands as through the window"); o.w8(RTC_DATA + 3, 0x77); - assert_eq!(run.dev.read8(at).data, o.bridge.read8(at).data); - assert_eq!(run.dev.read16(at).data, o.bridge.read16(at).data); - assert_eq!(run.dev.read32(at).data, o.bridge.read32(at).data); - assert_eq!(run.dev.read64(at).data, o.bridge.read64(at).data); - assert_eq!(run.dev.read32(at).data, 0x7777_7777); + assert_eq!(run.1.read8(at).data, o.bridge.read8(at).data); + assert_eq!(run.1.read16(at).data, o.bridge.read16(at).data); + assert_eq!(run.1.read32(at).data, o.bridge.read32(at).data); + assert_eq!(run.1.read64(at).data, o.bridge.read64(at).data); + assert_eq!(run.1.read32(at).data, 0x7777_7777); // The window aimed at PCI space nothing decodes: no run, and the // Bridge's master time-out is what answers there. o.set(reg::device(SLOT_IOC3), 0x1000 | (0x3000_0000 >> 20)); - assert!(runs.lock().iter().all(|r| !(r.start <= win && win < r.end))); + assert!(runs.lock().iter().all(|r| !r.0.contains(&win))); } diff --git a/src/ip30_machine.rs b/src/ip30_machine.rs index 7b434e1f9..62be9c897 100644 --- a/src/ip30_machine.rs +++ b/src/ip30_machine.rs @@ -39,6 +39,7 @@ use crate::ip30::{ widget_small_base, CachedFlashLines, Ip30Bus, LAST_PC, WIDGET_BRIDGE, WIDGET_HEART, WIDGET_SMALL_SIZE, WIDGET_XBOW, }; +use crate::pci::{BoardIrq, DevFn}; use crate::chunk_store::{get_chunks_as_words, put_words_as_chunks, ChunkStore}; use crate::snapshot::{field, ChunksManifest, Manifest, Snapshot, StateTable, SCHEMA_VERSION}; use crate::traits::{BusDevice, Device, MachineEvent, Resettable, Saveable}; @@ -220,7 +221,7 @@ impl OctaneMachine { // The onboard Bridge: memory and interrupts through HEART, the boot // flash on its first chip select, the system board's record on its // MicroLAN. - let bridge = Bridge::new(Upstream::single(crate::ip30::HEART_HOST, mem.clone(), heart.clone())); + let bridge = Bridge::new(Upstream::single(crate::ip30::HEART_HOST, mem.clone(), heart.clone()), &crate::ip30::PCI0); bridge.set_revision(if octane2 { BRIDGE_REVISION_OCTANE2 } else { BRIDGE_REVISION }); // The flash holds the PROM's own records as well as its code; with // `[octane] flash` they last from run to run. @@ -247,15 +248,17 @@ impl OctaneMachine { for s in &scsi { s.set_heartbeat(heartbeat.clone()); } - bridge.attach(0, scsi[0].clone()); - bridge.attach(1, scsi[1].clone()); + let pci = bridge.bus(); + let slot = |device| DevFn { device, function: 0 }; + pci.attach(slot(0), scsi[0].clone()); + pci.attach(slot(1), scsi[1].clone()); let ioc3 = Arc::new(Ioc3::octane(mac, false)); ioc3.set_heartbeat(heartbeat.clone()); let eth = Ioc3Eth::new(mac); eth.set_heartbeat(heartbeat.clone()); ioc3.attach_eth(eth.clone()); - bridge.attach(2, ioc3.clone()); - ioc3.connect_superio_interrupt(bridge.pin_line(SUPERIO_PIN)); + pci.attach(slot(2), ioc3.clone()); + ioc3.connect_superio_interrupt(pci.board_irq(BoardIrq(SUPERIO_PIN))); ioc3.set_timer_manager(timer_manager.clone()); // The guest switching the machine off through the clock. ioc3.set_event_sender(event_tx.clone()); @@ -271,12 +274,12 @@ impl OctaneMachine { if !cfg.no_audio { rad.enable_output(cfg.audio.clone()); } - bridge.attach(3, rad.clone()); + pci.attach(slot(3), rad.clone()); let virtio = cfg.virtio_serial.enabled.then(|| { let dev = Arc::new(crate::dev::virtio::pci::VirtioPci::new( crate::dev::virtio::console::from_config(&cfg.virtio_serial), )); - bridge.attach(VIRTIO_SLOT, dev.clone()); + pci.attach(slot(VIRTIO_SLOT), dev.clone()); eprintln!("iris: virtio-serial on Bridge slot {VIRTIO_SLOT}"); dev }); @@ -302,6 +305,7 @@ impl OctaneMachine { // switch. phys.map(widget_small_base(WIDGET_HEART), widget_small_base(WIDGET_HEART) + WIDGET_SMALL_SIZE, heart.clone(), "heart widget"); phys.attach_widget(WIDGET_BRIDGE, bridge.clone(), false); + phys.attach_long_window(WIDGET_BRIDGE, bridge.long_window()); if let Some(g) = &gfx { phys.attach_widget(WIDGET_GFX, g.clone(), false); } @@ -332,9 +336,10 @@ impl OctaneMachine { })); // Device-window pages a PCI BAR decodes whole go straight to the // function. - let weak = Arc::downgrade(&phys); - bridge.on_bar_runs(Box::new(move |runs| { - if let Some(p) = weak.upgrade() { + let (weak, bridge_weak) = (Arc::downgrade(&phys), Arc::downgrade(&bridge)); + bridge.bus().on_bar_mappings(Box::new(move |runs| { + if let (Some(p), Some(b)) = (weak.upgrade(), bridge_weak.upgrade()) { + let runs = runs.iter().map(|r| (r.cpu..r.cpu + r.len, b.bar_device(r))).collect(); p.set_widget_runs(WIDGET_BRIDGE, runs); } })); @@ -395,33 +400,25 @@ impl OctaneMachine { iris_hostcall::set_display(Arc::new(crate::dev::mgras::ImpactScreen(g.mgras().clone()))); } - // The shadow cache and the JIT read RAM straight from the ppmem - // window; descriptor writes lay it out again on the CPU thread. + // The shadow cache and the JIT read RAM through a cached view of the + // ppmem window, one per CPU; descriptor writes lay the window and + // every view out again on the CPU thread. if let Some(sp) = phys.phys.ppmem_space() { - use crate::ppmem::MappedMemory; - unsafe { - sp.set_bitmap_sink(cpu.ppmem_bitmap_ptr()); - cpu.set_tcache_window(sp.window_base()); - cpu.set_tcache_shift(crate::ppmem::window_shift(sp.window_size())); - let bitmap = cpu.tcache_bitmap_ptr(); - if !bitmap.is_null() { - sp.set_bitmap_sink2(bitmap); - } - #[cfg(feature = "jitv2")] - cpu.set_tcache_gen_window(sp.gen_window_base()); - for c in cpus.iter().skip(1) { - sp.add_bitmap_sink(c.ppmem_bitmap_ptr()); - c.set_tcache_window(sp.window_base()); + for c in cpus.iter() { + let v = sp + .add_cached_view(&[c.ppmem_bitmap_ptr(), c.tcache_bitmap_ptr()]) + .map_err(|e| format!("ppmem: a CPU's cached view: {e}"))?; + unsafe { + c.set_tcache_window(sp.view_base(v)); + c.set_cached_view(sp, v); c.set_tcache_shift(crate::ppmem::window_shift(sp.window_size())); - let bitmap = c.tcache_bitmap_ptr(); - if !bitmap.is_null() { - sp.add_bitmap_sink(bitmap); - } #[cfg(feature = "jitv2")] - c.set_tcache_gen_window(sp.gen_window_base()); + { + use crate::ppmem::MappedMemory; + c.set_tcache_gen_window(sp.gen_window_base()); + } } } - phys.phys.resync_ppmem_bitmap(); } // Publish the descriptors as they stand into the sinks just set. ram.set_memcfg(heart.memcfg()); @@ -457,6 +454,8 @@ impl OctaneMachine { monitor.register_device(crate::devlog::init_devlog()); monitor.register_device(Arc::new(crate::cpu::cpu_monitor::CpuMonitor::new(cpus.clone()))); monitor.register_device(heart.clone()); + // `pci`: the Bridge's bus, as the O2's MACE shows its own. + monitor.register_device(bridge.clone()); monitor.register_device(phys.phys.clone()); monitor.register_device(Arc::new(Ip30Monitor { phys: phys.clone(), @@ -932,9 +931,7 @@ impl OctaneMachine { if m.host_arch != std::env::consts::ARCH { return Err(format!("snapshot host_arch '{}' does not match current host '{}'", m.host_arch, std::env::consts::ARCH)); } - if m.schema_version != SCHEMA_VERSION { - return Err(format!("snapshot {name} is schema {}; this build reads Octane snapshots of schema {SCHEMA_VERSION}", m.schema_version)); - } + crate::snapshot::check_schema(name, m.schema_version)?; // As the Indy and the O2: build, CPU and disks must match. crate::snapshot::check_provenance(&m, cpu.model_name(), cpu.cache_kind(), &self.disks)?; let sv = m.schema_version; diff --git a/src/ip32.rs b/src/ip32.rs index 1069ae454..4f823f427 100644 --- a/src/ip32.rs +++ b/src/ip32.rs @@ -122,6 +122,27 @@ pub use crate::dev::mace_uart::*; pub use crate::dev::ns16550::*; pub use crate::dev::mace_id::*; +use crate::pci::{DevFn, IntxRouting}; + +/// MACE's PCI bus 0; the outputs are CRIME inputs. Source: the MACE +/// specification (Tables 85 and 87, pp. 164 and 166; Table 90, p. 179), which +/// gives the routing the software uses. No published document shows how the +/// board connects INTB-D of the option slot. +pub static PCI0: IntxRouting = IntxRouting { + devices: 32, + irqs: 16, + intx: &[ + (1, [Some(8), None, None, None]), // internal SCSI + (2, [Some(9), None, None, None]), // external SCSI + (3, [Some(10), Some(13), Some(14), Some(15)]), // INTB-D shared + // The specification routes device 4; no O2 board populates it. Using + // it for virtio-serial is iris's choice of a spare number, not + // hardware use. + (4, [Some(11), None, None, None]), + ], + board_irqs: &[], +}; + pub const RAM_WINDOW_SIZE: u32 = 8 * 128 * 1024 * 1024; /// CRIME's no-ECC alias of the memory window: the same memory as @@ -406,8 +427,10 @@ pub struct Ip32Bus { /// The bus everything is mapped on: what the CPU and the DMA masters use. pub phys: Arc, /// The runs of the PCI windows a BAR decodes whole, mapped over MACE - /// (`MacePci::on_bar_runs`). - pci_runs: Mutex>, + /// (`PciBus::on_bar_mappings`): where each starts and ends. + pci_runs: Mutex>, + /// The devices made for MACE's BAR mappings, once each. + bar_devices: crate::pci::BarDevices, pub(crate) ram: Arc, pub(crate) flash: O2Flash, /// Where to keep the flash between runs, if anywhere. Never the image the @@ -460,6 +483,7 @@ impl Ip32Bus { let mut bus = Self { phys: phys.clone(), pci_runs: Mutex::new(Vec::new()), + bar_devices: Default::default(), ram: ram.clone(), flash: O2Flash { bytes: Mutex::new(prom), writes: Mutex::new(Vec::new()) }, flash_file: opts.flash_file.clone(), @@ -471,7 +495,7 @@ impl Ip32Bus { gbe.tr = crime.tr.clone(); gbe }, - mace: Mace::new(ram.clone(), crime, ust), + mace: Mace::new(ram.clone(), crime, ust, &PCI0), misses: Arc::new(Misses::default()), scsi: Mutex::new(Vec::new()), }; @@ -498,8 +522,9 @@ impl Ip32Bus { let scsi_b = crate::dev::aic7880::Aic7880::new_at(1, bus_b); scsi.set_heartbeat(bus.mace.id.heartbeat.clone()); scsi_b.set_heartbeat(bus.mace.id.heartbeat.clone()); - bus.mace.pci.attach(IP32_SCSI_SLOT_B, "aic7880-b", scsi_b.clone()); - bus.mace.pci.attach(IP32_SCSI_SLOT, "aic7880", scsi.clone()); + let pci = bus.mace.pci.bus(); + pci.attach(DevFn { device: IP32_SCSI_SLOT_B, function: 0 }, scsi_b.clone()); + pci.attach(DevFn { device: IP32_SCSI_SLOT, function: 0 }, scsi.clone()); let mut list = bus.scsi.lock().unwrap(); list.push(scsi); list.push(scsi_b); @@ -521,7 +546,7 @@ impl Ip32Bus { bus.map(); // Window pages a PCI BAR decodes whole go straight to the function. let weak = Arc::downgrade(&bus); - bus.mace.pci.on_bar_runs(Box::new(move |runs| { + bus.mace.pci.bus().on_bar_mappings(Box::new(move |runs| { if let Some(b) = weak.upgrade() { b.set_pci_runs(runs); } @@ -531,16 +556,19 @@ impl Ip32Bus { /// MACE says which runs of its PCI windows a BAR decodes whole: map /// them over MACE, giving the runs it named before back to it. - fn set_pci_runs(&self, new: Vec) { + fn set_pci_runs(&self, new: Vec) { + use crate::pci::ByteOrder; let mut runs = self.pci_runs.lock().unwrap(); - for r in runs.iter() { + for &(start, end) in runs.iter() { // SAFETY: MACE is a field of this bus, alive as long as it is. - unsafe { self.phys.map_borrowed(r.start, r.end, &self.mace, "mace") }; + unsafe { self.phys.map_borrowed(start, end, &self.mace, "mace") }; } for r in &new { - self.phys.map(r.start, r.end, r.dev.clone(), "pci bar"); + // MACE's three windows share one set of order, so a run needs + // nothing from the window it lies in. + self.phys.map(r.cpu, r.cpu + r.len, self.bar_devices.get(r, 0xffff_ffff, ByteOrder::values(4)), "pci bar"); } - *runs = new; + *runs = new.iter().map(|r| (r.cpu, r.cpu + r.len)).collect(); } /// Put everything where CRIME and MACE decode it. The devices are fields @@ -667,14 +695,18 @@ impl crate::traits::Saveable for Ip32Bus { t.insert("mace".into(), self.mace.save_state()); let scsi = self.scsi.lock().unwrap().iter().map(|c| c.save_state()).collect(); t.insert("scsi".into(), toml::Value::Array(scsi)); - if let Some(words) = self.ram.stack_page_words() { - t.insert("stack_page".into(), toml::Value::Array(words.into_iter().map(crate::snapshot::hex_u64).collect())); - } toml::Value::Table(t) } fn load_state(&self, v: &toml::Value) -> Result<(), String> { use crate::snapshot::get_field; + // Older snapshots taken while POST's stack stood kept the page the + // uncached path saw. The stack lives in the CPU's cached view now, + // and they hold no cache lines to rebuild it from. + if get_field(v, "stack_page").is_some() { + return Err("this snapshot was taken during POST by an older iris, which kept the \ + stack page outside the cache; take it again".into()); + } if let Some(m) = get_field(v, "mace") { self.mace.load_state(m)?; } @@ -696,10 +728,9 @@ impl crate::traits::Saveable for Ip32Bus { c.load_state(s)?; } } - if let Some(toml::Value::Array(a)) = get_field(v, "stack_page") { - let words: Option> = a.iter().map(crate::snapshot::toml_u64).collect(); - self.ram.restore_stack_page(&words.ok_or("stack_page: not a list of dwords")?)?; - } + // MACE loaded before CRIME and the controllers: tell CRIME the pin + // levels now that it can take them. + self.mace.pci.bus().redrive(); Ok(()) } } @@ -710,7 +741,6 @@ impl Ip32Bus { /// on the hardware. The PROM sets up the rest again as it starts. pub fn power_on(&self) { use crate::traits::Resettable; - self.ram.set_stack_page(false); self.crime.regs.power_on(); let m = &self.mace; let devices: [&dyn Resettable; 8] = diff --git a/src/ip32/tests.rs b/src/ip32/tests.rs index 7031e5185..8f93271f0 100644 --- a/src/ip32/tests.rs +++ b/src/ip32/tests.rs @@ -214,6 +214,18 @@ fn a_bus_saves_and_loads_whole() { std::fs::remove_dir_all(&dir).ok(); } +/// A snapshot an older iris took during POST kept the stack page outside +/// the cache, which nothing can put back now: it is refused. +#[test] +fn a_snapshot_with_the_old_stack_page_is_refused() { + use crate::traits::Saveable; + let bus = Ip32Bus::new(16 << 20, vec![0u8; PROM_SIZE as usize]); + let mut v = bus.save_state(); + v.as_table_mut().unwrap().insert("stack_page".into(), toml::Value::Array(vec![])); + let e = bus.load_state(&v).unwrap_err(); + assert!(e.contains("take it again"), "{e}"); +} + /// Setting CONTROL's cold-reset bit asks the machine for a reset once, and /// the bit reads back clear, as the hardware clears it. #[test] @@ -940,12 +952,19 @@ fn nvram_survives_and_decodes_maces_spacing() { /// CLRINT, SCBPTR and the SCB array, which is exactly right. #[test] fn the_bridge_swaps_byte_lanes_for_a_little_endian_device() { - assert_eq!(MacePci::lane_swizzle(0x84, 1), 0x87, "HCNTRL"); - assert_eq!(MacePci::lane_swizzle(0x91, 1), 0x92, "CLRINT"); - assert_eq!(MacePci::lane_swizzle(0x93, 1), 0x90, "SCBPTR"); - assert_eq!(MacePci::lane_swizzle(0xbc, 1), 0xbf, "last byte of the SCB"); - assert_eq!(MacePci::lane_swizzle(0x84, 2), 0x86, "16-bit lanes swap in pairs"); - assert_eq!(MacePci::lane_swizzle(0x84, 4), 0x84, "a whole word needs no swap"); + use crate::pci::testing::TestCard; + use crate::pci::Bar; + let reached = |off: u32, size: usize| { + let card = TestCard::new([Bar::Mem(0x1000), Bar::Unused, Bar::Unused, Bar::Unused, Bar::Unused, Bar::Unused], 0); + crate::pci::ByteOrder::values(4).read(&*card, 0, off, size); + card.calls()[0].0 + }; + assert_eq!(reached(0x84, 1), 0x87, "HCNTRL"); + assert_eq!(reached(0x91, 1), 0x92, "CLRINT"); + assert_eq!(reached(0x93, 1), 0x90, "SCBPTR"); + assert_eq!(reached(0xbc, 1), 0xbf, "last byte of the SCB"); + assert_eq!(reached(0x84, 2), 0x86, "16-bit order swap in pairs"); + assert_eq!(reached(0x84, 4), 0x84, "a whole word needs no swap"); } /// Every O2 device answers the monitor through the bus: its command is @@ -971,3 +990,40 @@ fn every_device_answers_the_monitor_through_the_bus() { } assert!(bus.execute_command("nonesuch", &[], Box::new(std::io::sink())).is_err()); } + +// ---- the physical bus keeps what it maps ---- + +/// The physical bus keeps every device mapped on it for as long as it lives, +/// and every configuration write updates the BAR mappings, so a run must map +/// the same device each time or the list grows with every write. +#[test] +fn configuration_writes_do_not_make_the_physical_bus_keep_more_devices() { + use crate::pci::testing::TestCard; + use crate::pci::Bar; + let bus = Ip32Bus::new(1 << 20, vec![0u8; PROM_SIZE as usize]); + let pci = bus.mace.pci.bus(); + // A card in the specification's spare slot, with one memory BAR. + let at = DevFn { device: 4, function: 0 }; + let bars = [Bar::Mem(0x1000), Bar::Unused, Bar::Unused, Bar::Unused, Bar::Unused, Bar::Unused]; + pci.attach(at, TestCard::new(bars, 0)); + pci.config_write(at, 0x10, 4, 0x8000_2000); + pci.config_write(at, 0x04, 2, crate::pci::PCI_COMMAND_MEMORY); + let kept = bus.phys.kept_len(); + for i in 0..1000u32 { + pci.config_write(at, 0x3c, 1, i & 0xff); // interrupt line + } + assert_eq!(bus.phys.kept_len(), kept, "1000 interrupt-line writes"); + // A BAR that moves and comes back maps what it mapped before. + let place = |base: u32| { + pci.config_write(at, 0x10, 4, base); + }; + place(0x8000_4000); + let moved = bus.phys.kept_len(); + assert!(moved > kept, "a new placement maps new devices (one per window it appears in)"); + for _ in 0..50 { + place(0x8000_2000); + place(0x8000_4000); + } + assert_eq!(bus.phys.kept_len(), moved, "both placements are already mapped"); +} + diff --git a/src/ip32_machine.rs b/src/ip32_machine.rs index 1eb5c95b2..9cab747ca 100644 --- a/src/ip32_machine.rs +++ b/src/ip32_machine.rs @@ -240,27 +240,26 @@ impl O2Machine { let boot = Arc::new(BootState::new()); let memory = Arc::new(crate::ip32::memory_map()); let cpu: Arc = crate::with_cpu!(part, true, true, C => - Self::build_cpu::(memory.clone(), sysad, &timer_manager, arcs_fpu, count_hz, boot.clone(), bus.ram.clone(), part.prid())); + Self::build_cpu::(memory.clone(), sysad, &timer_manager, arcs_fpu, count_hz, boot.clone(), part.prid())); - // ppmem: the CPU's cache and the JIT reach RAM through the memory's - // window, publishing the mapped bitmap into the CPU's own fields, as - // the Indigo2 does. Remaps happen on the CPU thread (a CRIME bank - // control write), which is the window's contract. + // ppmem: the CPU's cache and the JIT reach RAM through a cached view + // of the memory's window, publishing its bitmap into the CPU's own + // fields, as the Indigo2 does. Remaps happen on the CPU thread (a + // CRIME bank control write), which is the window's contract. { - use crate::ppmem::MappedMemory; let sp = bus.phys.ppmem_space().ok_or("ppmem: the O2 needs its memory window")?; + let v = sp + .add_cached_view(&[cpu.ppmem_bitmap_ptr(), cpu.tcache_bitmap_ptr()]) + .map_err(|e| format!("ppmem: the CPU's cached view: {e}"))?; unsafe { - sp.set_bitmap_sink(cpu.ppmem_bitmap_ptr()); - cpu.set_tcache_window(sp.window_base()); - let bitmap = cpu.tcache_bitmap_ptr(); - if !bitmap.is_null() { - sp.set_bitmap_sink2(bitmap); - } + cpu.set_tcache_window(sp.view_base(v)); + cpu.set_cached_view(sp, v); #[cfg(feature = "jitv2")] - cpu.set_tcache_gen_window(sp.gen_window_base()); + { + use crate::ppmem::MappedMemory; + cpu.set_tcache_gen_window(sp.gen_window_base()); + } } - // The bus's direct paths read the same bitmap. - bus.phys.resync_ppmem_bitmap(); } // After the windows above: each compile worker takes the cache's @@ -361,7 +360,7 @@ impl O2Machine { } fn build_cpu(memory: Arc, sysad: Arc, tm: &Arc, arcs_fpu: bool, count_hz: u64, boot: Arc, - ram: Arc, prid: u32) -> Arc + prid: u32) -> Arc where C: crate::cpu::mips_cache_v2::CpuModel + MipsCache + From> + 'static, MipsCpu: CpuDevice, @@ -369,7 +368,6 @@ impl O2Machine { let cfg = MipsCpuConfig::for_model::(memory).with_prid(prid); let tlb = MipsTlb::new(cfg.tlb_entries); let mut exec: MipsExecutor = MipsExecutor::new(sysad, tlb, &cfg); - exec.cache.set_pd_tag_store_hook(Box::new(move |tag| ram.stack_tag_store(tag))); exec.core.set_timer_manager(tm.clone()); // Count runs at half the pipeline clock on both CPUs. exec.core.set_count_hz(count_hz.max(1)); @@ -641,7 +639,7 @@ impl O2Machine { fn read_snapshot(&self, name: &str) -> Result { use crate::chunk_store::{get_chunks_as_words, ChunkStore}; - use crate::snapshot::{Snapshot, SCHEMA_VERSION}; + use crate::snapshot::Snapshot; let snap = Snapshot::new(std::path::PathBuf::from("saves").join(name)); let m = snap.read_manifest()?.ok_or_else(|| format!("saves/{name}: no snapshot.toml"))?; if m.machine.as_deref() != Some(SNAPSHOT_MACHINE) { @@ -650,15 +648,7 @@ impl O2Machine { if m.host_arch != std::env::consts::ARCH { return Err(format!("snapshot host_arch '{}' does not match current host '{}'", m.host_arch, std::env::consts::ARCH)); } - if m.schema_version > SCHEMA_VERSION { - return Err(format!("snapshot schema_version {} is newer than this iris build supports ({SCHEMA_VERSION})", m.schema_version)); - } - if m.schema_version < SCHEMA_VERSION { - return Err(format!( - "snapshot {name} is schema {}; the O2's UART state changed shape in schema 5, so it cannot be restored", - m.schema_version - )); - } + crate::snapshot::check_schema(name, m.schema_version)?; // As the Indy: build, CPU and disks must match. crate::snapshot::check_provenance(&m, self.cpu.model_name(), self.cpu.cache_kind(), &self.disks)?; let sv = m.schema_version; diff --git a/src/ip35.rs b/src/ip35.rs index 432d8f497..48f9e64bf 100644 --- a/src/ip35.rs +++ b/src/ip35.rs @@ -1,18 +1,43 @@ -//! The IP35 processor bus: windows over small MMIO apertures. +//! The IP35 processor bus: Bedrock's four XKPHYS access-space views. //! -//! Maps replace whole 4 KiB pages as Physical does, while each endpoint still -//! rejects padding and enforces its cell permission. +//! Raw address bits 58:57 select Hub-special, I/O, fetch operations or uncached +//! RAM. Each view has a small fixed set of MMIO apertures, decoded by sorted +//! windows rather than a full physical-address table. Maps replace whole 4 KiB +//! pages as Physical does, while each endpoint still rejects padding and +//! enforces its cell permission. +//! +//! Cached RAM alone uses Physical, including its direct bank pointers, ppmem +//! and generation mappings. The MMIO views and register sub-bus need none of +//! those RAM facilities. Giving each its own full table would consume several +//! entries in the process-wide fault-handler registry per machine. //! //! Ported from hw/mips/sgi/sn1-node.c in jamesbraid/qemu-sgi-origami. +use crate::dev::bedrock_mem::{BANK_SPAN, FITTED_BANK}; use crate::physical::Physical; use crate::traits::*; #[cfg(feature = "jitv2")] -use std::sync::atomic::AtomicU64; +use std::sync::{atomic::AtomicU64, OnceLock}; use std::sync::Arc; +pub(crate) const CPU_PHYS_MASK: u64 = ((1u64 << 40) - 1) | (3u64 << 57); const PA_MASK: u64 = (1u64 << 40) - 1; +/// Where the Fuel has memory: four 1 GiB slots Bedrock's memory controller +/// can place its two 256 MiB banks in, depending on how the PROM programs +/// it, and the reset PROM between the first two. Uncached fetches of the +/// PROM's own code land in its range. +pub fn memory_map() -> crate::memory_map::MemoryMap { + use crate::memory_map::{MemKind, MemRange, MemoryMap}; + const NAMES: [&str; 4] = ["ram slot 0", "ram slot 1", "ram slot 2", "ram slot 3"]; + let mut ranges: Vec = NAMES.iter().enumerate().map(|(slot, &name)| { + let base = slot as u64 * BANK_SPAN; + MemRange::new(name, MemKind::Ram, base, base + FITTED_BANK) + }).collect(); + ranges.push(MemRange::new("prom", MemKind::Rom, 0x1fc0_0000, 0x2000_0000)); + MemoryMap::new(ranges) +} + struct Unclaimed; impl BusDevice for Unclaimed {} @@ -177,3 +202,143 @@ impl BusDevice for SparseWindowMap { sparse_access!(read64, write64, u64, BusRead64); } +/// The four uncached processor views decode MMIO separately from cached RAM. +pub struct Ip35Bus { + pub phys: Arc, + views: [SparseWindowMap; 4], +} +impl Ip35Bus { + pub fn new(phys: Arc) -> Self { + phys.set_default(Arc::new(Unclaimed), "IP35 unclaimed cached address"); + Self { + phys, + views: std::array::from_fn(|_| SparseWindowMap::default()), + } + } + /// Raw XKPHYS bits 58:57 select Hub-special, I/O, fetchop, or uncached RAM. + pub fn map(&mut self, raw_uc: u8, base: u64, size: u64, device: Arc) { + assert!(raw_uc < 4); + self.views[raw_uc as usize].map(base, size, device); + } + pub fn set_fallback(&mut self, raw_uc: u8, end: u64, device: Arc) { + self.map(raw_uc, 0, end, device); + } + fn target(&self, a: u64, n: u64) -> Option<(&SparseWindowMap, u64)> { + let pa = plain_address(a, n)?; + Some((&self.views[(a >> 57) as usize], pa)) + } + pub fn cached_view(&self, plain_memory: Arc) -> Arc { + Arc::new(CachedPhysical { + phys: self.phys.clone(), + plain_memory, + }) + } +} +fn plain_address(a: u64, n: u64) -> Option { + if a & !CPU_PHYS_MASK != 0 { + return None; + } + let pa = a & PA_MASK; + (pa.checked_add(n)? <= PA_MASK + 1).then_some(pa) +} +struct CachedPhysical { + phys: Arc, + plain_memory: Arc, +} +impl CachedPhysical { + fn memory_target(&self, a: u64, n: u64) -> Option<(&dyn BusDevice, u64)> { + self.target(a, n).map(|(physical, pa)| { + // MD owns every guest write to a possible fitted-RAM prefix, so + // all paths update its stable logical generation counters. PROM + // and other devices retain the cached Physical connection. + let memory: &dyn BusDevice = if pa / BANK_SPAN < 4 && pa % BANK_SPAN < FITTED_BANK { + self.plain_memory.as_ref() + } else { physical }; + (memory, pa) + }) + } + fn target(&self, a: u64, n: u64) -> Option<(&Physical, u64)> { + // UC selectors belong to the uncached SysAd connection. Reject them + // here rather than turning a malformed cached request into an alias. + if a & !PA_MASK != 0 || a & (n - 1) != 0 { + return None; + } + Some((self.phys.as_ref(), plain_address(a, n)?)) + } +} +macro_rules! processor_access { + ($r:ident,$w:ident,$t:ty,$out:ident,$n:expr,$writes:ident) => { + fn $r(&self, a: u64) -> $out { + self + .target(a, $n) + .map(|(v, pa)| v.$r(pa)) + .unwrap_or_else($out::err) + } + fn $w(&self, a: u64, v: $t) -> u32 { + self + .$writes(a, $n) + .map(|(view, pa)| view.$w(pa, v)) + .unwrap_or(BUS_ERR) + } + }; +} +impl BusDevice for Ip35Bus { + fn write64_masked(&self, a: u64, val: u64, mask: u64) -> u32 { + if a & 7 != 0 { + return BUS_ERR; + } + self.target(a, 8) + .map(|(view, pa)| view.write64_masked(pa, val, mask)) + .unwrap_or(BUS_ERR) + } + fn write_narrow_from(&self, a: u64, size: usize, gpr: u64) -> u32 { + self.target(a, size as u64) + .map(|(view, pa)| view.write_narrow_from(pa, size, gpr)) + .unwrap_or(BUS_ERR) + } + processor_access!(read8, write8, u8, BusRead8, 1, target); + processor_access!(read16, write16, u16, BusRead16, 2, target); + processor_access!(read32, write32, u32, BusRead32, 4, target); + processor_access!(read64, write64, u64, BusRead64, 8, target); +} +impl BusDevice for CachedPhysical { + fn write64_masked(&self, address: u64, value: u64, mask: u64) -> u32 { + self.memory_target(address, 8).map(|(view, pa)| view.write64_masked(pa, value, mask)) + .unwrap_or(BUS_ERR) + } + fn mem_ptr(&self, a: u64) -> Option<*const u64> { + self.target(a, 8).and_then(|(view, pa)| view.mem_ptr(pa)) + } + #[cfg(feature = "jitv2")] + fn gen_ptr(&self, a: u64) -> *const AtomicU64 { + self.target(a, 1).map(|(view, pa)| { + let generation = self.plain_memory.gen_ptr(pa); + if generation.is_null() { view.gen_ptr(pa) } else { generation } + }).unwrap_or(std::ptr::null()) + } + fn read_block(&self, a: u64, out: &mut [u64]) -> u32 { + self.target(a, 8) + .map(|(view, pa)| view.read_block(pa, out)).unwrap_or(BUS_ERR) + } + fn write_block(&self, a: u64, data: &[u64]) -> u32 { + self.memory_target(a, 8) + .map(|(view, pa)| view.write_block(pa, data)).unwrap_or(BUS_ERR) + } + processor_access!(read8, write8, u8, BusRead8, 1, memory_target); + processor_access!(read16, write16, u16, BusRead16, 2, memory_target); + processor_access!(read32, write32, u32, BusRead32, 4, memory_target); + processor_access!(read64, write64, u64, BusRead64, 8, memory_target); + fn dma_read64(&self, a: u64) -> BusRead64 { + plain_address(a, 8) + .map(|pa| self.plain_memory.dma_read64(pa)) + .unwrap_or_else(BusRead64::err) + } + fn dma_write64(&self, a: u64, v: u64) -> u32 { + plain_address(a, 8) + .map(|pa| self.plain_memory.dma_write64(pa, v)) + .unwrap_or(BUS_ERR) + } +} + +#[cfg(test)] +mod tests; diff --git a/src/ip35/tests.rs b/src/ip35/tests.rs new file mode 100644 index 000000000..e0b1b98f1 --- /dev/null +++ b/src/ip35/tests.rs @@ -0,0 +1,764 @@ +use super::*; +use crate::traits::{BusDevice, BUS_OK}; + +struct Word(u64); +impl BusDevice for Word { + fn read64(&self, _: u64) -> BusRead64 { + BusRead64::ok(self.0) + } +} + +#[derive(Clone, Debug, PartialEq, Eq)] +enum NarrowWriteCall { + Byte(u64, u8), + Half(u64, u16), + From(usize, u64, u64), +} +/// Records every narrow write it sees. With `sees_register` it overrides +/// `write_narrow_from` and keeps the whole source register, as Bedrock's ECC +/// view does; otherwise it takes the trait's default. +#[derive(Default)] +struct NarrowWriteSpy { + sees_register: bool, + calls: std::sync::Mutex>, +} +impl NarrowWriteSpy { + fn new(sees_register: bool) -> Arc { + Arc::new(Self { sees_register, ..Default::default() }) + } + fn calls(&self) -> Vec { + self.calls.lock().unwrap().clone() + } +} +impl BusDevice for NarrowWriteSpy { + fn write8(&self, address: u64, value: u8) -> u32 { + self.calls.lock().unwrap().push(NarrowWriteCall::Byte(address, value)); + BUS_OK + } + fn write16(&self, address: u64, value: u16) -> u32 { + self.calls.lock().unwrap().push(NarrowWriteCall::Half(address, value)); + BUS_OK + } + fn write_narrow_from(&self, address: u64, size: usize, gpr: u64) -> u32 { + if !self.sees_register { + return if size == 1 { self.write8(address, gpr as u8) } else { self.write16(address, gpr as u16) }; + } + self.calls.lock().unwrap().push(NarrowWriteCall::From(size, address, gpr)); + BUS_OK + } +} + +/// An uncached SB or SH reaches a device with the whole source register, at the +/// device-local address; only a device that overrides the call sees more than +/// the low bits, and a window outside every map is a bus error. +/// The Fuel's processor is the R14000 at revision 2.3 and 500 MHz, built as +/// the R12000 type with its PRId set over it; the Octane's own part is +/// revision 2.4. +#[test] +fn the_fuel_runs_the_r14000_at_revision_2_3() { + use crate::cpu::mips_cache_shadow::R12000ShadowCache; + use crate::cpu::mips_cache_v2::CpuModel as Model; + use crate::cpu::parts; + use crate::ip35_machine::CPUS; + assert_eq!(CPUS, &[&parts::R14000_500]); + let part = CPUS[0]; + assert_eq!((part.mhz, part.rev, part.l2_kb), (500, (2, 3), 2048)); + assert_eq!(part.prid(), 0x0f23); + assert_eq!(part.count_hz(), 250_000_000); + // `with_cpu!` builds the type `FuelCpu` names for this part. + let built = crate::with_cpu!(part, true, false, C => std::any::type_name::()); + assert_eq!(built, std::any::type_name::()); + // Typed R12000, it takes the R10000 family's behaviour: reserved + // secondary-tag bits read as zero, and a refused uncached store posts. + assert!(::R10K_FAMILY); + assert_ne!(::SECONDARY_TAG_MASK, u64::MAX); + assert!(::POSTS_UNCACHED_STORES); + assert!(::PARTIAL_LWR_KEEPS_UPPER); + // The executor carries the chip's PRId. + // The executor is large; the default test stack is not. + std::thread::Builder::new() + .stack_size(64 << 20) + .spawn(|| { + let cpu = crate::ip35_machine::build_cpu(Arc::new(Ip35Bus::new(Physical::new(Vec::new()))), crate::ip35_machine::CPUS[0]); + assert_eq!(cpu.core.cp0_prid, 0x0f23); + }) + .unwrap() + .join() + .expect("thread panicked"); +} + +#[test] +fn narrow_stores_carry_the_register_to_devices_that_ask_for_it() { + let mut bus = Ip35Bus::new(Physical::new(Vec::new())); + let plain = NarrowWriteSpy::new(false); + let ecc = NarrowWriteSpy::new(true); + bus.map(3, 0x1000, 0x1000, plain.clone()); + bus.map(3, 0x2000, 0x1000, ecc.clone()); + + let source = 0x1122_3344_5566_77a5u64; + let uc = 3u64 << 57; + assert_eq!(bus.write_narrow_from(uc | 0x1003, 1, source), BUS_OK); + assert_eq!(bus.write_narrow_from(uc | 0x1004, 2, source), BUS_OK); + assert_eq!( + plain.calls(), + vec![NarrowWriteCall::Byte(3, source as u8), NarrowWriteCall::Half(4, source as u16)], + "an ordinary window gets the architectural low bits", + ); + + assert_eq!(bus.write_narrow_from(uc | 0x2004, 2, source), BUS_OK); + assert_eq!(ecc.calls(), vec![NarrowWriteCall::From(2, 4, source)]); + + assert_eq!(bus.write_narrow_from(uc | 0x8000, 2, source), BUS_ERR, "no window there"); +} + +/// Equal physical offsets in different XKPHYS access spaces reach different devices. +#[test] +fn uc_spaces_keep_equal_offsets_distinct() { + let mut bus = Ip35Bus::new(Physical::new(Vec::new())); + for space in 0..4 { + bus.map( + space, + 0x0100_0000, + 0x1000, + Arc::new(Word(0x100 + space as u64)), + ); + } + for space in 0..4 { + let value = bus.read64(((space as u64) << 57) | 0x0100_0000); + assert_eq!(value.status, BUS_OK); + assert_eq!(value.data, 0x100 + space as u64); + } +} + +/// The processor must retain both XKPHYS access-space bits through translation. +#[test] +fn xkphys_uc_bits_survive_the_board_cpu_mask() { + // MipsExecutor and the TLB's inline arrays exceed the test worker's stack. + std::thread::Builder::new() + .stack_size(4 * 1024 * 1024) + .spawn(|| { + let bus = Arc::new(Ip35Bus::new(Physical::new(Vec::new()))); + let mut cpu = crate::ip35_machine::build_cpu(bus, crate::ip35_machine::CPUS[0]); + cpu.core.write_cp0( + 12, + (cpu.core.cp0_status | crate::cpu::mips_core::STATUS_KX) as u64, + ); + for raw_uc in 0..4u64 { + let va = 0x9000_0000_0100_0020 | (raw_uc << 57); + for _ in 0..2 { + let translated = cpu.debug_translate(va); + assert_eq!(translated.status, crate::cpu::mips_exec::TR_UNCACHED); + assert_eq!(translated.phys, (raw_uc << 57) | 0x0100_0020); + } + } + }) + .expect("spawn") + .join() + .expect("thread panicked"); +} + +/// RAM placed above the first GiB must remain shared by cached, uncached and DMA paths. +#[test] +fn cached_and_uncached_memory_keep_high_addresses() { + let board = crate::ip35_machine::FuelBoard::new(&[], Arc::new(|| 0)).unwrap(); + assert_eq!(board.md.md_bus().write64(8, 0x33), BUS_OK); + let at = (1 << 30) + 0x1000; + assert_eq!(board.md.memory_view().write64(at, 0x12345678), BUS_OK); + assert_eq!(board.cached_bus().read64(at).data, 0x12345678); + assert_eq!(board.bus.read64((3 << 57) | at).data, 0x12345678); + assert_eq!(board.bus.read64(at).status, BUS_ERR); + assert_eq!(board.md.dma_view().read64(at).data, 0x12345678); +} + +/// Memory diagnostics must notify both processor interfaces and refuse bad uncached data. +#[test] +fn memory_errors_reach_both_interfaces_and_uncached_reads_fail() { + let board = crate::ip35_machine::FuelBoard::new(&[], Arc::new(|| 0)).unwrap(); + let correctable = 1 << 61; + board.pi[0].write64(0xb0, correctable); + board.md.md_bus().write64(8, 0x33); + for directory in [false, true] { + if directory { + board.md.md_bus().write64(0x40000, 2); + board.md.directory_bus().write64(0, 1 << 16); + board.md.md_bus().write64(0x40000, 0); + } else { + board.md.ecc_bus().write8(7, 1); + } + assert_eq!(board.bus.read64(0).status, BUS_OK); + for pi in &board.pi { + assert_eq!(pi.read64(0xa0).data & correctable, correctable); + } + assert_eq!(board.irq.bits() & (1 << 11), 1 << 11); + for pi in &board.pi { + pi.write64(0x90, 64 + 61); + assert_eq!(pi.read64(0xa0).data & correctable, 0); + } + assert_eq!(board.irq.bits() & (1 << 11), 0); + } + + let board = crate::ip35_machine::FuelBoard::new(&[], Arc::new(|| 0)).unwrap(); + board.md.md_bus().write64(8, 0x33); + board.md.ecc_bus().write8(7, 3); + for selector in [0, 3] { + assert_eq!(board.bus.read64(selector << 57).status, BUS_ERR); + } + assert_eq!(board.cached_bus().read64(0).status, BUS_OK); + assert_eq!(board.md.dma_view().read64(0).status, BUS_OK); +} + +/// Firmware resets the PCI link before reinitializing the attached storage controller. +#[test] +fn fuel_link_reset_resets_the_attached_scsi_engine() { + use crate::dev::isp1040::{mbox, reg, CTRL_INT_ENABLE, CTRL_RISC_INT_ENABLE, HOST_CMD_SHIFT}; + use crate::pci::PciFunction; + let board = crate::ip35_machine::FuelBoard::new(&[], Arc::new(|| 0)).unwrap(); + let isp = &board.scsi; + isp.state().config().write(1, 6); + isp.state().config().write(5, 0x02000000); + isp.bar_write(1, reg::BUS_CTRL, u32::from(CTRL_INT_ENABLE | CTRL_RISC_INT_ENABLE), 2); + isp.bar_write(1, reg::MAILBOX_BASE, u32::from(mbox::NOP), 2); + isp.bar_write(1, reg::HOST_CMD, 0x5 << HOST_CMD_SHIFT, 2); + assert_eq!(isp.bar_read(1, reg::MAILBOX_BASE, 2), u32::from(mbox::COMPLETE)); + assert_ne!(isp.bar_read(1, reg::BUS_CTRL, 2), 0); + assert_eq!(board.xbridge.widget(0).unwrap().write32(0x2f4, 1), BUS_OK); + assert_eq!(isp.state().config().command(), 0); + assert_eq!(isp.state().config().bar_base(1), 0); + assert_eq!(isp.bar_read(1, reg::BUS_CTRL, 2), 0); + assert_eq!(isp.bar_read(1, reg::MAILBOX_BASE, 2), 0); + assert_eq!(isp.bar_read(1, reg::MAILBOX_BASE + 2, 2), 0x4953); +} + +/// The PROM reads the station address from the L1 and IRIX from the +/// Ethernet; both must report the configured one. +#[test] +fn the_configured_station_address_reaches_the_l1_and_the_ethernet() { + let mac = [0x08, 0x00, 0x69, 0x21, 0x43, 0x65]; + let board = crate::ip35_machine::FuelBoard::with_config(crate::ip35_machine::FuelBoardConfig { + payload: &[], clock: Arc::new(|| 0), flash_path: None, + station_mac: mac, + timekeeper: Arc::new(crate::dev::timekeeper::Timekeeper::fuel().unwrap()), + ps2_input: true, usb_input: true, + l1: Default::default(), + graphics: crate::config::GraphicsBoard::None, + graphics_pool: (2, 32), + graphics_raster: (crate::config::RasterUnit::Cpu, "auto"), + }).unwrap(); + assert_eq!(board.l1.station_mac(), mac); + assert_eq!(board.eth.mac(), mac); +} + +/// The L1's reported revision and brick serial keep their defaults unless +/// the board is built with replacements. +#[test] +fn the_l1_identity_overrides_reach_the_controller() { + use crate::ip35_machine::{FuelBoard, FuelBoardConfig, L1Identity}; + let build = |l1| FuelBoard::with_config(FuelBoardConfig { + payload: &[], clock: Arc::new(|| 0), flash_path: None, + station_mac: [8, 0, 0x69, 0x12, 0x34, 0x56], + timekeeper: Arc::new(crate::dev::timekeeper::Timekeeper::fuel().unwrap()), + ps2_input: true, usb_input: false, l1, + graphics: crate::config::GraphicsBoard::None, + graphics_pool: (2, 32), + graphics_raster: (crate::config::RasterUnit::Cpu, "auto"), + }).unwrap().l1.config(); + let stock = build(L1Identity::default()); + assert_eq!(stock.revision, Some([1, 48, 1])); + assert_eq!(stock.brick_serial.as_deref(), Some("EMUN00")); + let set = build(L1Identity { revision: Some([2, 3, 4]), brick_serial: Some("EMUN09".into()) }); + assert_eq!(set.revision, Some([2, 3, 4])); + assert_eq!(set.brick_serial.as_deref(), Some("EMUN09")); +} + +fn fuel_dma_board() -> crate::ip35_machine::FuelBoard { + let board = crate::ip35_machine::FuelBoard::new(&[], Arc::new(|| 0)).unwrap(); + assert_eq!(board.md.md_bus().write64(8, 0x33), BUS_OK); + board.xbridge.head(15).write32(0x24, 15); + board.xbridge.set_wide_slot_mask(15, 0); + board +} + +fn fuel_dma_address(address: u64) -> u64 { + (10 << 60) | (1 << 55) | address +} + +/// Storage DMA buffers can begin unaligned and cross a physical page boundary. +#[test] +fn dma_preserves_byte_order_and_crosses_ragged_pages() { + use crate::pci::PciDmaStatus; + let board = fuel_dma_board(); + let dma = board.xbridge.dma(15, 0); + let memory = board.md.memory_view(); + for (address, length) in [(0x1030, 8), (0x1103, 27), (0x1ffb, 19)] { + let data: Vec = (0..length).map(|i| 0x31 + i as u8).collect(); + assert_eq!(dma.dma_write(fuel_dma_address(address), &data), PciDmaStatus::Ok); + for (i, &byte) in data.iter().enumerate() { + assert_eq!(memory.read8(address + i as u64).data, byte); + } + let mut back = vec![0xff; length]; + assert_eq!(dma.dma_read(fuel_dma_address(address), &mut back), PciDmaStatus::Ok); + assert_eq!(back, data); + } + // Without byte swapping enabled, the bridge reverses each 32-bit word. + assert_eq!(dma.dma_write((10 << 60) | 0x3000, &[0, 1, 2, 3, 4, 5, 6, 7]), PciDmaStatus::Ok); + assert_eq!(memory.read64(0x3000).data, 0x0302010007060504); +} + +/// A failed DMA read must clear only refused chunks and preserve completed data. +#[test] +fn dma_refusals_clear_failed_reads_without_losing_completed_chunks() { + use crate::pci::PciDmaStatus; + let board = fuel_dma_board(); + let dma = board.xbridge.dma(15, 0); + let memory = board.md.memory_view(); + // Bank zero has 256 MiB fitted. The second qword is outside its backing. + let last = 0x0fff_fff8; + assert_eq!(memory.write64(last, 0x0102030405060708), BUS_OK); + let mut bytes = [0xff; 16]; + assert_eq!(dma.dma_read(fuel_dma_address(last), &mut bytes), PciDmaStatus::MasterAbort); + assert_eq!(bytes, [1, 2, 3, 4, 5, 6, 7, 8, 0, 0, 0, 0, 0, 0, 0, 0]); + assert_eq!(dma.dma_write(fuel_dma_address(last), &[0x42; 16]), PciDmaStatus::MasterAbort); + assert_eq!(memory.read64(last).data, 0x4242424242424242); + board.io.disable_device(15, 0); + bytes.fill(0xff); + assert_eq!(dma.dma_read(fuel_dma_address(last), &mut bytes), PciDmaStatus::MasterAbort); + assert_eq!(bytes, [0; 16]); + assert_eq!(dma.dma_write(fuel_dma_address(last), &[0x99; 8]), PciDmaStatus::MasterAbort); + assert_eq!(memory.read64(last).data, 0x4242424242424242); +} + +/// A wide DMA leg must report an uncorrectable word once without publishing bad bytes. +#[test] +fn fuel_dma_checks_an_aligned_uncorrectable_word_once() { + use crate::dev::bedrock_md::MemoryError; + use crate::pci::PciDmaStatus; + use std::sync::atomic::{AtomicU64, Ordering}; + let board = fuel_dma_board(); + assert_eq!(board.md.memory_view().write64(0, 0), BUS_OK); + assert_eq!(board.md.ecc_bus().write8(7, 3), BUS_OK); + let count = Arc::new(AtomicU64::new(0)); + let seen = count.clone(); + let md = Arc::downgrade(&board.md); + board.md.on_error(Arc::new(move |kind| { + assert_eq!(kind, MemoryError::Uncorrectable); + assert_eq!(md.upgrade().unwrap().current_cell(), u64::MAX); + seen.fetch_add(1, Ordering::Relaxed); + })); + // The former callback read the same persistent UCE once for each byte. + for address in 0..8 { + assert_eq!(board.io.inbound_read(15, Some(0), address, 1).status, BUS_OK); + } + assert_eq!(count.swap(0, Ordering::Relaxed), 8); + assert_ne!(board.md.md_bus().read64(0x60018).data & (1 << 61), 0); + let mut bytes = [0xff; 8]; + assert_eq!(board.xbridge.dma(15, 0).dma_read(fuel_dma_address(0), &mut bytes), PciDmaStatus::Ok); + assert_eq!(bytes, [0; 8]); + assert_eq!(count.load(Ordering::Relaxed), 1); + let record = board.md.md_bus().read64(0x60010).data; + assert_ne!(record & (1 << 60), 0); + assert_eq!(record & (1 << 61), 0); + assert_eq!(board.md.uncached_view().read64(0).status, BUS_ERR); +} + +/// Inbound Hub writes must preserve access width and the requesting device identity. +#[test] +fn dma_hub_access_preserves_width_and_source_permissions() { + use crate::pci::PciDmaStatus; + let board = fuel_dma_board(); + let dma = board.xbridge.dma(15, 0); + let hub = (1 << 47) | 0x1800000; + for (interface, offset) in [(0, 0), (1, 0x200000)] { + let address = fuel_dma_address(hub + offset + 0x90); + assert_eq!(dma.dma_write(address, &0x10cu64.to_be_bytes()), PciDmaStatus::Ok); + assert_ne!(board.pi[interface].read64(0x98).data & (1 << 12), 0); + for width in [1, 2, 4] { + assert_eq!(dma.dma_write(address, &[0; 8][..width]), PciDmaStatus::MasterAbort); + assert_ne!(board.pi[interface].read64(0x98).data & (1 << 12), 0); + } + assert_eq!(dma.dma_write(address, &12u64.to_be_bytes()), PciDmaStatus::Ok); + assert_eq!(board.pi[interface].read64(0x98).data & (1 << 12), 0); + } + let address = fuel_dma_address(hub + 0x90); + // I/O modify must retain PI region protection and requesting-slot identity. + assert_eq!(board.pi[0].write64(0x10, 0), BUS_OK); + assert_eq!(dma.dma_write(address, &0x10cu64.to_be_bytes()), PciDmaStatus::MasterAbort); + assert_eq!(board.pi[0].read64(0x98).data & (1 << 12), 0); + assert_eq!(board.pi[0].write64(0x10, 1), BUS_OK); + board.io.disable_device(15, 0); + assert_eq!(dma.dma_write(address, &0x10cu64.to_be_bytes()), PciDmaStatus::MasterAbort); + assert_eq!(board.pi[0].read64(0x98).data & (1 << 12), 0); + assert_eq!(board.xbridge.dma(15, 1).dma_write(address, &0x10cu64.to_be_bytes()), PciDmaStatus::Ok); + assert_ne!(board.pi[0].read64(0x98).data & (1 << 12), 0); + let mut bytes = [0xff; 8]; + assert_eq!(board.xbridge.dma(15, 1).dma_read( + fuel_dma_address(hub + 0x40), &mut bytes, + ), PciDmaStatus::MasterAbort); + assert_eq!(bytes, [0; 8]); + assert_eq!(board.bus.read64((1 << 57) | 0x1800040).status, BUS_OK); +} + +/// Rounded MMIO pages must not expose padding or bypass cell permissions. +#[test] +fn exact_windows_reject_padding_and_revoked_permissions() { + use std::sync::atomic::{AtomicBool, Ordering}; + struct Offset; + impl BusDevice for Offset { + fn read64(&self, offset: u64) -> BusRead64 { BusRead64::ok(offset) } + fn write64(&self, _offset: u64, _value: u64) -> u32 { BUS_OK } + } + let permitted = Arc::new(AtomicBool::new(true)); + let permission = permitted.clone(); + let window = Window::new(0x1000, 0x88, Arc::new(Offset)) + .permitted(Arc::new(move || permission.load(Ordering::Acquire))); + assert_eq!(window.read64(0x1080).data, 0x80); + assert_eq!(window.read64(0x1088).status, BUS_ERR); + assert_eq!(window.read64(0x1084).status, BUS_ERR); + assert_eq!(window.read64(0x0ff8).status, BUS_ERR); + assert_eq!(window.read64(u64::MAX).status, BUS_ERR); + permitted.store(false, Ordering::Release); + assert_eq!(window.read64(0x1000).status, BUS_ERR); + assert_eq!(window.write64(0x1000, 1), BUS_ERR); + permitted.store(true, Ordering::Release); + assert_eq!(window.write64(0x1000, 1), BUS_OK); +} + +/// Board default setup must preserve the direct RAM path and its shared DMA contents. +#[test] +fn direct_cached_banks_remain_visible_after_board_defaults_are_installed() { + let board = crate::ip35_machine::FuelBoard::new(&[], Arc::new(|| 0)).unwrap(); + assert_eq!(board.md.md_bus().write64(8, 0x33), BUS_OK); + let cached = board.cached_bus(); + assert!(cached.mem_ptr(0x2000).is_some()); + assert_eq!(cached.write64(0x2000, 0x123456789abcdef0), BUS_OK); + assert_eq!(board.md.dma_view().read64(0x2000).data, 0x123456789abcdef0); + assert!(cached.mem_ptr(3 << 57).is_none(), "UC selectors cannot alias cached RAM"); +} + +/// A device overlay replaces its pages, including padding, while its neighbors +/// keep their original address translation. +#[test] +fn sparse_overlays_keep_neighbors_and_refuse_the_new_windows_padding() { + let mut map = SparseWindowMap::default(); + map.map(0x1000, 0x4000, Arc::new(Word(1))); + map.map(0x2040, 0x80, Arc::new(Word(2))); + assert_eq!(map.read64(0x1000).data, 1); + assert_eq!(map.read64(0x2040).data, 2); + assert_eq!(map.read64(0x20b8).data, 2); + assert_eq!(map.read64(0x20bc).status, BUS_ERR); + assert_eq!(map.read64(0x2000).status, BUS_ERR); + assert_eq!(map.read64(0x20c0).status, BUS_ERR); + assert_eq!(map.read64(0x3000).data, 1); + assert_eq!(map.read64(0x5000).status, BUS_ERR); + map.map(0x1000, 0x4000, Arc::new(Word(3))); + assert_eq!(map.ranges.len(), 1); + assert_eq!(map.read64(0x2040).data, 3); +} + +/// Sparse MMIO holes remain bus errors and a revoked cell permission does not +/// fall through to an older mapping beneath the protected window. +#[test] +fn sparse_holes_and_revoked_permissions_never_expose_older_devices() { + use std::sync::atomic::{AtomicBool, Ordering}; + let permission = Arc::new(AtomicBool::new(true)); + let allowed = permission.clone(); + let mut map = SparseWindowMap::default(); + map.map(0, 0x3000, Arc::new(Word(1))); + map.map(0x1000, 8, Arc::new(Window::new(0, 8, Arc::new(Word(2))) + .permitted(Arc::new(move || allowed.load(Ordering::Acquire))))); + assert_eq!(map.read64(0x1000).data, 2); + permission.store(false, Ordering::Release); + assert_eq!(map.read64(0x1000).status, BUS_ERR); + assert_eq!(map.read64(0x1008).status, BUS_ERR); + assert_eq!(map.read64(0x2000).data, 1); + assert_eq!(map.read64(0x3000).status, BUS_ERR); +} + +/// Code-page lookup receives plain physical addresses and reads them through +/// the cached plane. Cached stores must also update the MD view. +#[cfg(feature = "jitv2")] +#[test] +fn code_generations_use_the_stable_ram_endpoint() { + struct Generations { + ram_only: bool, + generation: AtomicU64, + writes: std::sync::Mutex>, + } + impl Generations { + fn new(ram_only: bool) -> Arc { + Arc::new(Self { ram_only, generation: AtomicU64::new(0), writes: Default::default() }) + } + } + impl BusDevice for Generations { + fn gen_ptr(&self, address: u64) -> *const AtomicU64 { + if !self.ram_only || address < 0x1000 { &self.generation } else { std::ptr::null() } + } + fn write64(&self, address: u64, value: u64) -> u32 { + self.writes.lock().unwrap().push((address, value)); + BUS_OK + } + fn write64_masked(&self, address: u64, value: u64, mask: u64) -> u32 { + self.write64(address, value & mask) + } + fn write_block(&self, address: u64, values: &[u64]) -> u32 { + for (index, value) in values.iter().enumerate() { + self.write64(address + index as u64 * 8, *value); + } + BUS_OK + } + } + let physical = Physical::new(Vec::new()); + let bus = Ip35Bus::new(physical.clone()); + let raw_ram = Generations::new(false); + let stable_ram = Generations::new(true); + let rom = Generations::new(false); + map_device(&physical, 0, 0x1000, raw_ram.clone(), "raw RAM generation"); + map_device(&physical, 0x1fc00000, 0x1000, rom.clone(), "ROM generation"); + let cached = bus.cached_view(stable_ram.clone()); + assert_eq!(cached.gen_ptr(0x40), &stable_ram.generation as *const AtomicU64); + assert_eq!(cached.gen_ptr(0x1fc00000), &rom.generation as *const AtomicU64); + assert_eq!(cached.write64(0x40, 0x1234), BUS_OK); + assert_eq!(cached.write_block(0x48, &[0x5678]), BUS_OK); + assert_eq!(cached.write64_masked(0x50, 0xaabb, 0xff), BUS_OK); + assert_eq!(*stable_ram.writes.lock().unwrap(), [(0x40, 0x1234), (0x48, 0x5678), (0x50, 0xbb)]); + assert_eq!(cached.write64(0x1fc00000, 0x4321), BUS_OK); + assert_eq!(*rom.writes.lock().unwrap(), [(0, 0x4321)], "the ROM window translates to a device-local offset"); + assert!(raw_ram.writes.lock().unwrap().is_empty()); +} + +/// The JIT's page lookup and inline compile read code through the CPU's cached +/// bus, which a board whose uncached connection decodes plain addresses as +/// something else sets to the view the cache fills from. The compile pool is +/// started on that same object. +#[test] +fn the_cpu_reads_code_through_the_cached_view_not_the_uncached_connection() { + // An executor is large; a test thread's stack is not. + std::thread::Builder::new().stack_size(64 << 20).spawn(cpu_cached_bus).unwrap().join().unwrap(); +} + +fn cpu_cached_bus() { + let board = crate::ip35_machine::FuelBoard::new(&[], Arc::new(|| 0)).unwrap(); + let cpu = board.build_cpu(crate::ip35_machine::CPUS[0]); + let cached = board.cached_bus(); + assert!(Arc::ptr_eq(cpu.cached_bus(), &cached)); + let sysad: Arc = board.bus.clone(); + assert!(!Arc::ptr_eq(cpu.cached_bus(), &sysad), "sysad is the uncached connection"); +} + +/// Unmodeled Hub heads retain the PROM compatibility readback without +/// absorbing the adjacent, independently implemented L1 UART window. +#[test] +fn compatibility_hub_heads_do_not_hide_the_l1_uart_window() { + let board = crate::ip35_machine::FuelBoard::new(&[], Arc::new(|| 0)).unwrap(); + assert_ne!(board.md.current_cell() & 1, 0); + for base in [0x1000_0000, 0x3000_0000] { + for offset in [0, 8, 0x40, 0x78] { + let address = base + offset; + assert_eq!(board.bus.write8(address, 0xa5), BUS_OK); + assert_eq!(board.bus.write16(address, 0xa5a5), BUS_OK); + assert_eq!(board.bus.write32(address, 0xa5a5a5a5), BUS_OK); + assert_eq!(board.bus.write64(address, 0x2000), BUS_OK); + let value = board.bus.read8(address); + assert_eq!(value.status, BUS_OK); + assert_eq!(value.data, 0); + let value = board.bus.read16(address); + assert_eq!(value.status, BUS_OK); + assert_eq!(value.data, 0); + let value = board.bus.read32(address); + assert_eq!(value.status, BUS_OK); + assert_eq!(value.data, 0); + let value = board.bus.read64(address); + assert_eq!(value.status, BUS_OK); + assert_eq!(value.data, 0); + } + assert_eq!(board.bus.read64(base + 0x80).status, BUS_OK); + } +} + +/// A busy host port or unusable transcript path must not prevent a serial +/// machine from starting, including when any of its three ports serves CI. +#[test] +fn unavailable_serial_ports_and_logs_do_not_abort_the_machine() { + use crate::config::{FuelConsole, MachineConfig}; + let listeners: Vec<_> = (0..3).map(|_| std::net::TcpListener::bind(("127.0.0.1", 0)).unwrap()).collect(); + let ports: Vec<_> = listeners.iter().map(|socket| socket.local_addr().unwrap().port()).collect(); + let board = crate::ip35_machine::FuelBoard::new(&[], Arc::new(|| 0)).unwrap(); + let mut config = MachineConfig::default(); + config.fuel.l1_port = Some(ports[0]); + config.serial_port_a = Some(ports[1]); + config.serial_port_b = Some(ports[2]); + config.ci = true; + let missing = std::env::temp_dir().join(format!("iris-fuel-missing-log-{}", std::process::id())).join("absent").join("serial.txt"); + assert!(!missing.parent().unwrap().exists(), "the log parent must be absent"); + config.serial_log = Some(missing.to_string_lossy().into_owned()); + config.fuel.l1_log = config.serial_log.clone(); + for (console, selection) in [FuelConsole::L1, FuelConsole::Ioc3A, FuelConsole::Ioc3B].into_iter().enumerate() { + config.fuel.console = selection; + crate::ip35_machine::FuelMachine::connect_serial_for_test(&config, &board, console).unwrap(); + } + config.ci = false; + crate::ip35_machine::FuelMachine::connect_serial_for_test(&config, &board, 0).unwrap(); +} + +/// The Fuel's OHCI root ports each carry a low-speed device, and host +/// input reaches them through the IOC3's PS/2 controller object; the USB +/// pair is fitted only when asked for. +#[test] +fn the_fuel_plugs_a_usb_keyboard_and_mouse_only_when_configured_on() { + use crate::pci::PciFunction; + let ports = |usb_input| { + let board = crate::ip35_machine::FuelBoard::with_config(crate::ip35_machine::FuelBoardConfig { + payload: &[], clock: Arc::new(|| 0), flash_path: None, + station_mac: [8, 0, 0x69, 0x12, 0x34, 0x56], + timekeeper: Arc::new(crate::dev::timekeeper::Timekeeper::fuel().unwrap()), + ps2_input: true, usb_input, + l1: Default::default(), + graphics: crate::config::GraphicsBoard::None, + graphics_pool: (2, 32), + graphics_raster: (crate::config::RasterUnit::Cpu, "auto"), + }).unwrap(); + // Power each port: one with a device shows it connected. + board.usb.bar_write(0, 0x54, 0x100, 4); + board.usb.bar_write(0, 0x58, 0x100, 4); + let status = [0x54, 0x58].map(|at| board.usb.bar_read(0, at, 4)); + (board, status) + }; + let (board, status) = ports(true); + for s in status { + assert_eq!(s & 1, 1, "device connected"); + assert_eq!(s & 0x200, 0x200, "low speed"); + } + // The mouse the USB side fits makes the shared controller report a mouse + // only once the guest has configured it. + assert!(!board.ioc3.ps2().mouse_enabled()); + let (_, status) = ports(false); + assert_eq!(status.map(|s| s & 1), [0, 0], "nothing plugged in"); +} + +fn fuel_with_graphics(graphics: crate::config::GraphicsBoard) -> crate::ip35_machine::FuelBoard { + crate::ip35_machine::FuelBoard::with_config(crate::ip35_machine::FuelBoardConfig { + payload: &[], clock: Arc::new(|| 0), flash_path: None, + station_mac: [0x08, 0x00, 0x69, 0x21, 0x43, 0x65], + timekeeper: Arc::new(crate::dev::timekeeper::Timekeeper::fuel().unwrap()), + ps2_input: true, usb_input: true, + l1: Default::default(), + graphics, + graphics_pool: (2, 32), + graphics_raster: (crate::config::RasterUnit::Cpu, "auto"), + }).unwrap() +} + +/// The VPro's link status in the crossbar, in its slot's data half. +const VPRO_LINK_STATUS: u64 = 0x100 + 5 * 0x40 + 0x10 + 4; +/// A processor access to widget 13 through Bedrock's small window. +const VPRO_PIO: u64 = 13 << 24; + +// The VPro cases follow tests/qtest/sn1-fuel-vpro-test.c in +// jamesbraid/qemu-sgi-origami: widget-identity, profile-identity-and-routing +// and headless-fuel-has-no-board. + +// VPro host-interface registers the tests below touch, as byte offsets in +// the board's widget space. The board's own constants are private to it. +const ODSY_IDENTITY: u64 = 0x00; +const ODSY_CONTROL: u64 = 0x20; +const ODSY_CONTROL_WIDGET_MASK: u32 = 0xf; +const ODSY_INT_DEST_LOWER: u64 = 0x38; +const ODSY_INTR_CLEAR: u64 = 0x101c; +const ODSY_INTR_ENABLE_SET: u64 = 0x1034; +const ODSY_INTR_FIFO_LOW_WATER: u32 = 0x200; +const ODSY_ROUTE_DEST_UPPER: u64 = 0x1044; +const ODSY_ROUTE_DEST_LOWER: u64 = 0x104c; +const ODSY_ROUTE_VECTORS: u64 = 0x1054; +const ODSY_ROUTE_DEST_WIDGET_SHIFT: u32 = 25; + +/// A Fuel IRIX inventory lists its VPro under xtalk/13: the processor must +/// reach the board there, learn its widget number from it, and find the +/// link alive in the crossbar. +#[test] +fn the_vpro_answers_at_widget_13_with_its_link_alive() { + for (graphics, revision) in [(crate::config::GraphicsBoard::V12, 0xa), (crate::config::GraphicsBoard::V10, 0xa)] { + let board = fuel_with_graphics(graphics); + let identity = board.io.pio_read(VPRO_PIO | (ODSY_IDENTITY + 4), 4); + assert_eq!(identity.status, BUS_OK); + assert_eq!(identity.data as u32, (revision << 28) | 0x0c01_3047, "{graphics:?}"); + let control = board.io.pio_read(VPRO_PIO | (ODSY_CONTROL + 4), 4).data as u32; + assert_eq!(control & ODSY_CONTROL_WIDGET_MASK, 13); + assert_ne!(board.xbridge.crossbar().read32(VPRO_LINK_STATUS).data & (1 << 31), 0); + } +} + +/// The monitor's `vpro` commands resolve on a Fuel that has a VPro, and are +/// unknown on one that has none. +#[test] +fn the_monitor_resolves_vpro_commands_on_a_fuel_with_a_vpro() { + let resolves = |board: &crate::ip35_machine::FuelBoard, command: &str| { + board.monitor_devices().iter().any(|d| d.register_commands().iter().any(|(c, _)| c == command)) + }; + let with = fuel_with_graphics(crate::config::GraphicsBoard::V12); + assert!(resolves(&with, "vpro"), "a Fuel with a V12 answers `vpro`"); + assert!(resolves(&with, "ps2"), "the other board commands stay"); + let without = fuel_with_graphics(crate::config::GraphicsBoard::None); + assert!(!resolves(&without, "vpro"), "a Fuel with no VPro has no `vpro`"); + assert!(resolves(&without, "ps2")); +} + +/// The board's interrupt packets cross the XBridge to Bedrock as the PCI +/// heads' do: a raised source sets its vector's pending bit in the +/// processor interface the destination names, and a cleared one clears it. +#[test] +fn vpro_interrupt_packets_set_and_clear_a_processor_interface_pending_bit() { + let board = fuel_with_graphics(crate::config::GraphicsBoard::V12); + // Register space (bit 47), the node's hub window and PI0's pending + // modify register, at the Bedrock's crossbar port 0xa. + let packet = (1u64 << 47) | 0x180_0000 | 0x90; + let vector = 5; + let write = |offset: u64, value: u64| { + assert_eq!(board.io.pio_write(VPRO_PIO | offset, 4, value), BUS_OK, "{offset:#x}"); + }; + write(ODSY_ROUTE_DEST_UPPER, (0xa << ODSY_ROUTE_DEST_WIDGET_SHIFT) | (packet >> 32)); + write(ODSY_ROUTE_DEST_LOWER, packet & 0xffff_ffff); + write(ODSY_ROUTE_VECTORS, vector); + write(ODSY_INTR_ENABLE_SET, ODSY_INTR_FIFO_LOW_WATER as u64); + let pending = || board.pi[0].read64(0x98).data & (1 << vector); + assert_ne!(pending(), 0); + write(ODSY_INTR_CLEAR, ODSY_INTR_FIFO_LOW_WATER as u64); + assert_eq!(pending(), 0); +} + +/// A link reset of the graphics port resets the board behind it, as the +/// PCI port's does for the functions behind its head. +#[test] +fn a_graphics_link_reset_resets_the_vpro() { + let board = fuel_with_graphics(crate::config::GraphicsBoard::V12); + let destination = VPRO_PIO | (ODSY_INT_DEST_LOWER + 4); + assert_eq!(board.io.pio_write(destination, 4, 0x55), BUS_OK); + assert_eq!(board.io.pio_read(destination, 4).data, 0x55); + let link_reset = 0x100 + 5 * 0x40 + 0x30 + 4; + assert_eq!(board.xbridge.crossbar().write32(link_reset, 1), BUS_OK); + assert_eq!(board.io.pio_read(destination, 4).data, 0); +} + +/// Without graphics nothing answers at widget 13 and its link is down, as +/// on a headless Fuel. +#[test] +fn a_fuel_without_graphics_has_no_widget_13() { + let board = fuel_with_graphics(crate::config::GraphicsBoard::None); + assert!(board.vpro.is_none()); + assert_ne!(board.io.pio_read(VPRO_PIO | 4, 4).status, BUS_OK); + assert_eq!(board.xbridge.crossbar().read32(VPRO_LINK_STATUS).data & (1 << 31), 0); +} + +/// IRIX's VPro driver writes each processor interface's graphics credit +/// counter at attach and tears the board down if it reads back zero; the +/// registers must keep what the guest writes, per interface. +#[test] +fn processor_interface_graphics_registers_keep_their_writes() { + let board = fuel_with_graphics(crate::config::GraphicsBoard::V12); + // Slot A's credit counter and bias; PI1 keeps its own block. + assert_eq!(board.pi[0].write64(0x308, 100), BUS_OK); + assert_eq!(board.pi[0].write64(0x310, 0x1000 | 68), BUS_OK); + assert_eq!(board.pi[0].read64(0x308).data, 100); + assert_eq!(board.pi[0].read64(0x310).data, 68); + assert_eq!(board.pi[1].read64(0x308).data, 0); + board.pi[0].power_on(); + assert_eq!(board.pi[0].read64(0x308).data, 0); +} diff --git a/src/ip35_machine.rs b/src/ip35_machine.rs new file mode 100644 index 000000000..2a400ba9a --- /dev/null +++ b/src/ip35_machine.rs @@ -0,0 +1,1244 @@ +//! The Fuel (IP34 board, IP35 software platform): an R14000 on Bedrock, +//! XBridge and the base I/O devices, with Iris's CPU thread, monitor and serial console. +//! +//! Selected with `[machine] profile = "fuel_ip34"`. Shared keys select the +//! PROM, NVRAM, memory and SCSI devices; `[fuel]` selects the serial console +//! and writable flash. `ip35::Ip35Bus` maps the processor views over the +//! same Physical memory banks. The IOC3 carries PS/2 keyboard and mouse ports +//! (default) and the OHCI controller can take a USB pair; without graphics the +//! machine runs on a serial console. With `[graphics] board = "v12"` (or +//! `"v10"`) a VPro board answers at crossbar widget 13, the Fuel's graphics +//! slot. +//! +//! # What is approximated here +//! +//! The two Hub-special head windows return zero on reads and acknowledge +//! writes without side effects. PROM 6.210 writes the 64-bit value 0x2000 +//! at relative offsets 0 and 0x40 before DRAM discovery. Their hardware +//! meaning is unknown; Bedrock logging records accesses to these windows. +//! +//! The processor interfaces' graphics flow-control registers store what +//! IRIX writes, which is what its VPro driver checks, but count no stores: +//! no uncached store into the graphics page takes a credit and the board +//! returns none, so the credit-starvation interrupt never fires. The board's +//! DMA reaches memory through Bedrock's inbound path as widget 13. +//! +//! Board composition ported from hw/mips/sgi/sn1-node.c +//! in jamesbraid/qemu-sgi-origami. + +use crate::config::{scsi_label, FuelConsole, GraphicsBoard, MachineConfig}; +use crate::cpu::mips_exec::{CpuDevice, MipsCpu, MipsCpuConfig, MipsExecutor}; +use crate::cpu::mips_tlb::MipsTlb; +use crate::dev::bedrock_flash::{Flash, PART_SIZE}; +use crate::dev::bedrock_io::{BedrockIo, SIZE as IO_SIZE}; +use crate::dev::bedrock_lb::{BedrockLb, SIZE as LB_SIZE}; +use crate::dev::bedrock_md::{BedrockMd, MemoryError}; +use crate::dev::bedrock_ni::{BedrockNi, SIZE as NI_SIZE}; +use crate::dev::bedrock_pi::{reg as pi_reg, BedrockPi, BedrockPiConfig, SIZE as PI_SIZE}; +use crate::dev::bedrock_xb::{BedrockXb, SIZE as XB_SIZE}; +use crate::dev::heart::{host_clock, Clock}; +use crate::dev::ioc3::{Ioc3, Ioc3Config}; +use crate::dev::odyssey::{Odyssey, PacketSender}; +use crate::dev::ioc3_eth::Ioc3Eth; +use crate::dev::isp12160::Isp12160; +use crate::dev::l1::{L1, L1Config, L1Event}; +use crate::dev::ohci::Ohci; +use crate::dev::usb_hid::{UsbKeyboard, UsbMouse}; +use crate::dev::prom_image; +use crate::dev::ps2::Ps2Controller; +use crate::dev::serial_host::{null_serial_backend, open_log, tcp_serial_backend, CiSerialBackend, LogMode, SerialBackend, TeeBackend}; +use crate::dev::timekeeper::Timekeeper; +use crate::dev::xbridge::{InterruptPacket, XBridge}; +use crate::pci::{BoardIrq, DevFn}; +use crate::devlog::{init_devlog, LogModule}; +use crate::gdb_stub::CpuDebug; +use crate::gfx_display::GfxDisplay; +use crate::hptimer::TimerManager; +use crate::cpu::mips_cache_shadow::R12000ShadowCache; +use crate::cpu::parts::{self, Part}; +use crate::cpu::mips_cache_v2::MipsCache; +use crate::ip35::{map_device, Ip35Bus, SparseWindowMap, Window, CPU_PHYS_MASK}; +use crate::irq::{InterruptController, IrqLine}; +use crate::machine_state::GuestState; +use crate::monitor::{Monitor, DEFAULT_PORT as DEFAULT_MONITOR_PORT}; +use crate::net::{GatewayConfig, Link, NatControl}; +use crate::perf_monitor::PerfMonitor; +use crate::ppmem::devmap::SLOT_SHIFT; +use crate::scsi::{attach_from_config, ScsiBus}; +use crate::traits::{BusDevice, BusRead8, BusRead16, BusRead32, BusRead64, Device, MachineEvent, Resettable, Signal, BUS_ERR, BUS_OK}; +use crate::workstation::{register_system_controller, DiscChanger, MachineRef, MachineSlot, Workstation}; +use parking_lot::Mutex; +use std::net::{Ipv4Addr, SocketAddr}; +use std::path::{Path, PathBuf}; +use std::sync::mpsc::{channel, Receiver}; +use std::sync::{atomic::{AtomicBool, AtomicU64, Ordering}, Arc}; + +pub struct FuelMachine { + board: Arc, + cpu: Arc, + timers: Arc, + monitor: Arc, + monitor_port: u16, + ci_serial: Option>, + console: usize, + ci: bool, + scratch: Option, + timekeeper: Arc, + nvram: String, + frames: Option>, + nat: Option<( + Arc, + (Ipv4Addr, Ipv4Addr, Ipv4Addr), + )>, + events: Option>, + slot: Option, + lifecycle: GuestState, +} + +impl FuelMachine { + pub fn new(cfg: &MachineConfig) -> Result { + cfg.validate()?; + let prom = prom_image::load(Path::new(&cfg.prom), 8 * 1024 * 1024) + .map_err(|e| format!("Fuel {e}"))?; + let timekeeper = Arc::new(Timekeeper::fuel_with_nvram_file( + Path::new(&cfg.nvram), + )?); + let board = Arc::new(FuelBoard::with_config(FuelBoardConfig { + payload: &prom, + clock: host_clock(), + flash_path: cfg.fuel.flash.as_deref().map(Path::new), + station_mac: cfg.network().mac, + timekeeper: timekeeper.clone(), + ps2_input: cfg.fuel.ps2_input, + usb_input: cfg.fuel.usb_input, + l1: L1Identity { + revision: cfg.fuel.l1_revision.as_deref().map(crate::dev::l1::parse_revision).transpose()?, + brick_serial: cfg.fuel.brick_serial.clone(), + }, + graphics: cfg.graphics_board(), + graphics_pool: (cfg.graphics.threads, cfg.graphics.tile), + graphics_raster: (cfg.graphics.raster, cfg.graphics.gpu_backend.as_str()), + })?); + let timers = Arc::new(TimerManager::new()); + let part = cfg.cpu(); + let mut exec = board.build_cpu(part); + // The CPU's Compare matches blink the graphics status bar's LED. + exec.core.fasttick_count = board.fasttick.clone(); + exec.core.set_timer_manager(timers.clone()); + exec.core.set_count_hz( + cfg.clock + .fixed_mhz + .map_or(part.count_hz(), |mhz| (mhz * 1_000_000.0) as u64) + .max(1), + ); + exec.core.count_write_samples = true; + let poll_board = board.clone(); + exec.batch_hook = Some(Box::new(move |_| poll_board.poll())); + let cpu: Arc = Arc::new(MipsCpu::<_, R12000ShadowCache>::new(*exec)); + board.status.set_cpu_cycles(cpu.cycles_ptr()); + board.status.set_count_hz(cpu.count_hz_atomic()); + if let Some(space) = board.md.physical().ppmem_space() { + // The CPU reads memory through a cached view of its own. The + // PROM keeps its stack in primary data lines before it has set + // up memory, so lines may stand in every bank slot without it. + let v = space + .add_cached_view(&[cpu.ppmem_bitmap_ptr(), cpu.tcache_bitmap_ptr()]) + .map_err(|e| format!("ppmem: the CPU's cached view: {e}"))?; + for slot in 0..4u64 { + let at = slot * crate::dev::bedrock_mem::BANK_SPAN; + space.allow_unbacked_lines(at, at + crate::dev::bedrock_mem::FITTED_BANK); + } + unsafe { + cpu.set_tcache_window(space.view_base(v)); + cpu.set_cached_view(space, v); + #[cfg(feature = "jitv2")] + cpu.set_tcache_gen_window(board.md.generation_base()); + } + board.md.rebuild_memory(); + } + // After the windows above: each compile worker takes the cache's + // geometry and the CPU model's policies once, as it starts, and + // without the windows the geometry has no inline load or store path. + // Workers read plain physical code addresses through the CPU's cached + // bus, which serves them; the uncached views would decode them as Hub + // registers. + #[cfg(feature = "jitv2")] + { + let cpu_device: Arc = cpu.clone().as_device(); + let jit = cpu.jitv2(); + let mut g = jit.lock(); + g.compile_queue.set_thread_count(cfg.jitv2.threads.max(1)); + g.compile_queue.set_cpu(Arc::downgrade(&cpu_device)); + g.compile_queue.set_owner(Arc::downgrade(&jit)); + // Not under lockstep, as on the other machines: lockstep compiles + // inline on the CPU thread, and a running pool both owns the + // Codegen inline needs and deadlocks the CPU thread's flush. + #[cfg(not(feature = "jitv2_lockstep"))] + { + let stats = g.stats.clone(); + g.compile_queue.start(board.cached_bus(), stats); + } + // The compile pool keeps only a weak handle on the CPU. + std::mem::forget(cpu_device); + } + board.pi[0].set_interrupt_controller(Arc::new(CpuIrq(Arc::downgrade(&cpu)))); + let (tx, rx) = channel(); + let reset_tx = tx.clone(); + board.lb.connect_soft_reset(Arc::new(move || { + let _ = reset_tx.send(MachineEvent::HardReset); + })); + let reset_tx = tx.clone(); + board.ni.connect_local_reset(Arc::new(move || { + let _ = reset_tx.send(MachineEvent::HardReset); + })); + let weak_cpu = Arc::downgrade(&cpu); + board.l1.set_event_handler(Arc::new(move |event| { + match event { + L1Event::Power(false) | L1Event::Shutdown => { + let _ = tx.send(MachineEvent::PowerOff); + } + L1Event::Power(true) | L1Event::Reset => { + let _ = tx.send(MachineEvent::HardReset); + } + L1Event::SoftReset => { + if let Some(cpu) = weak_cpu.upgrade() { + cpu.as_device().signal(Signal::Reset(true)); + } + } + L1Event::Nmi => dlog!(LogModule::L1, "Fuel: NMI entry is not implemented by the CPU"), + } + })); + let scratch = Self::attach_scsi(cfg, &board)?; + let console = match cfg.fuel.console { + FuelConsole::L1 => 0, + FuelConsole::Ioc3A => 1, + FuelConsole::Ioc3B => 2, + }; + let ci_serial = cfg.ci.then(|| board.serial[console].clone()); + Self::connect_serial(cfg, &board, console)?; + let monitor = Monitor::new(); + monitor.register_device(init_devlog()); + monitor.register_device(Arc::new(crate::cpu::cpu_monitor::CpuMonitor::new(vec![cpu.clone()]))); + monitor.register_device(PerfMonitor::new( + cpu.running_flag(), cpu.cycles_ptr(), cpu.fasttick_count(), None, None, + )); + for device in board.monitor_devices() { + monitor.register_device(device); + } + cpu.register_locks(); + board.scsi.register_locks(); + board.ioc3.register_locks(); + board.xbridge.register_locks(); + board.flash.register_locks(); + board.md.register_locks(); + let (mut nat, mut frames) = (None, None); + let net = GatewayConfig::from_network(&cfg.network()); + if let Some(link) = Link::with_gateway(net.clone())? { + frames = Some(link.frames()); + nat = Some((link.nat(), (net.client_ip, net.gateway_ip, net.netmask))); + board.eth.attach(link); + } + eprintln!( + "iris: Fuel IP34, {}, 512 MiB, {}, serial console {:?}", + cfg.cpu().label(), + if board.vpro.is_some() { cfg.graphics_board().label() } else { "no graphics" }, + cfg.fuel.console + ); + Ok(Self { + board, + cpu, + timers, + monitor: Arc::new(monitor), + monitor_port: cfg.monitor_port.unwrap_or(DEFAULT_MONITOR_PORT), + ci_serial, + console, + ci: cfg.ci, + scratch, + timekeeper, + nvram: cfg.nvram.clone(), + nat, + frames, + events: Some(rx), + slot: None, + lifecycle: GuestState::Idle, + }) + } + + fn attach_scsi(cfg: &MachineConfig, board: &FuelBoard) -> Result, String> { + let attached = attach_from_config(cfg, Some("fuel"), &|channel| { + board.scsi.bus_for_channel(channel) + .ok_or_else(|| format!("controller {channel} (a Fuel has channels 0 and 1)")) + })?; + if let Some(device) = attached.daynaports.first() { + return Err(format!("{}: kind = \"daynaport\" is not available on the Fuel", + scsi_label(device.controller, device.id))); + } + Ok(attached.scratch) + } + + fn connect_serial( + cfg: &MachineConfig, + board: &FuelBoard, + console: usize, + ) -> Result<(), String> { + let [a, b] = cfg.serial_ports(); + let ports = [cfg.fuel.l1_port.unwrap_or(8882), a, b]; + if ports[0] == ports[1] || ports[0] == ports[2] || ports[1] == ports[2] { + return Err("Fuel L1 and IOC3 serial ports must be distinct".into()); + } + let labels = ["L1", "IOC3 A", "IOC3 B"]; + let settings = ["fuel.l1_port", "serial_port_a", "serial_port_b"]; + let mut log_config = cfg.clone(); + for (index, port) in ports.into_iter().enumerate() { + log_config.serial_log = if index == console { + cfg.serial_log.clone() + } else if index == 0 { + cfg.fuel.l1_log.clone() + } else { + None + }; + let tag = format!("Fuel {}", labels[index]); + let log = open_log(&log_config, &tag, LogMode::Fresh); + let backend: Arc = if cfg.ci && index == console { + if let Some(log) = log { + board.serial[index].set_log(log); + } + board.serial[index].clone() + } else { + let mut backend = null_serial_backend(); + match tcp_serial_backend(port) { + Some((socket, actual_port)) => { + eprintln!("iris: {tag} on 127.0.0.1:{actual_port}"); + backend = socket; + } + None => eprintln!( + "iris: {tag}: port {port} unavailable; is another iris using it? ({})", + settings[index] + ), + } + match log { + Some(log) => Arc::new(TeeBackend::new(backend, log)), + None => backend, + } + }; + if index == 0 { + board.l1.set_serial_backend(backend); + } else { + board.ioc3.set_serial_backend(index - 1, backend); + } + } + Ok(()) + } + + #[cfg(test)] + pub(crate) fn connect_serial_for_test(cfg: &MachineConfig, board: &FuelBoard, console: usize) -> Result<(), String> { + Self::connect_serial(cfg, board, console) + } + +} + +struct CpuIrq(std::sync::Weak); +impl InterruptController for CpuIrq { + fn set_line(&self, line: u32, level: bool) { + if line < 5 { + if let Some(cpu) = self.0.upgrade() { + cpu.as_device().signal(Signal::Interrupt(line + 2, level)); + } + } + } +} + +impl Workstation for FuelMachine { + fn start(&mut self) { + self.lifecycle = GuestState::Starting; + self.monitor + .clone() + .start_server(format!("127.0.0.1:{}", self.monitor_port)); + self.restart_peripherals(); + if !self.ci { + self.cpu_start(); + } + } + fn stop(&mut self) { + self.cpu.clone().as_device().stop(); + self.board.stop(); + if let Err(e) = self.timekeeper.flush_nvram() { + dlog!(LogModule::Bedrock, "Fuel NVRAM: {e}"); + } + if self.lifecycle != GuestState::Idle { + self.lifecycle = GuestState::Halted; + } + } + fn reset(&mut self) { + self.stop(); + self.cpu.power_on(); + for pi in &self.board.pi { + pi.power_on(); + } + self.board.md.power_on(); + self.board.lb.power_on(); + self.board.io.power_on(); + self.board.ni.power_on(); + self.board.xb.power_on(); + self.board.l1.power_on(); + self.board.flash.power_on(); + self.board.xbridge.power_on(); + self.board.ioc3.power_on(); + self.board.eth.power_on(); + self.board.scsi.power_on(); + self.board.usb.power_on(); + if let Some(vpro) = &self.board.vpro { + vpro.power_on(); + } + self.timekeeper.power_on(); + self.restart_peripherals(); + } + fn restart_peripherals(&mut self) { + self.board.start(); + } + fn cpu_start(&self) { + self.cpu.clone().as_device().start(); + } + fn cpu_is_running(&self) -> bool { + self.cpu.clone().as_device().is_running() + } + fn machine_slot(&mut self) -> MachineSlot { + let ptr = self as *mut Self as *mut dyn Workstation; + self.slot + .get_or_insert_with(|| Arc::new(Mutex::new(MachineRef(ptr)))) + .clone() + } + fn register_system_controller(&mut self) { + let slot = self.machine_slot(); + let events = self + .events + .take() + .expect("Fuel system controller registered twice"); + register_system_controller(&self.monitor, slot, Some(events)); + } + fn monitor_addr(&self) -> Option { + self.monitor.bound_addr() + } + fn lifecycle(&self) -> GuestState { + self.lifecycle + } + fn set_lifecycle(&mut self, state: GuestState) { + self.lifecycle = state; + } + fn get_display(&self) -> Option> { + self.board.vpro.as_ref().map(|vpro| vpro.display()) + } + fn register_monitor_device(&self, device: Arc) { + self.monitor.register_device(device); + } + fn get_ps2(&self) -> Arc { + self.board.ioc3.ps2() + } + fn get_timer_manager(&self) -> Arc { + self.timers.clone() + } + fn get_cpu_debug(&self) -> Arc { + self.cpu.clone().debug_adapter() + } + fn fasttick_count(&self) -> Arc { + self.cpu.fasttick_count() + } + fn get_ci_serial(&self) -> Option> { + self.ci_serial.clone() + } + fn inject_serial_console(&self, bytes: &[u8]) { + self.board.serial[self.console].push_host(bytes); + } + fn as_any_mut(&mut self) -> &mut dyn std::any::Any { + self + } + fn save_nvram(&self, path: Option<&str>) -> Result { + if path.is_some_and(|p| p != self.nvram) { + return Err("Fuel NVRAM save uses the configured path".into()); + } + self.timekeeper.flush_nvram()?; + Ok(self.nvram.clone()) + } + fn scratch_path(&self) -> Option<&Path> { + self.scratch.as_deref() + } + fn disc_changer(&self) -> Option> { + Some(self.board.scsi.clone() as Arc) + } + fn pending_chd_sync_count(&self) -> usize { + (0..2).map(|channel| self.board.scsi.bus_for_channel(channel).unwrap().pending_chd_sync_count()).sum() + } + fn flush_disks(&self) -> std::io::Result<()> { + self.board.flush_disks() + } + fn sync_chd_disks( + &self, + only: Option, + progress: &mut dyn FnMut(usize, usize, f32), + cancel: &dyn Fn() -> bool, + ) -> std::io::Result { + let buses: Vec<_> = (0..2).map(|channel| self.board.scsi.bus_for_channel(channel).unwrap()).collect(); + ScsiBus::sync_chd_disks(&buses, only, progress, cancel) + } + fn net_frames(&self) -> Option> { self.frames.clone() } + fn nat_control(&self) -> Option> { + self.nat.as_ref().map(|(n, _)| n.clone()) + } + fn nat_expected(&self) -> Option<(Ipv4Addr, Ipv4Addr, Ipv4Addr)> { + self.nat.as_ref().map(|(_, a)| *a) + } +} + +impl Drop for FuelMachine { + fn drop(&mut self) { + if let Some(slot) = self.slot.take() { + slot.lock().0 = MachineRef::null(); + } + self.stop(); + self.monitor.shutdown(); + } +} + +/// The processors a Fuel can be built with, the default first: the R14000A +/// the Fuel ships with. +pub const CPUS: &[&Part] = &[&parts::R14000_500]; + +/// The cache type `with_cpu!` builds for every entry of [`CPUS`]; a test pins +/// it. The board's devices are not generic over the processor, so it names the +/// type once here. +pub(crate) type FuelCpu = MipsExecutor; + +// The 500 MHz / 2 MiB PROM record selects these boot-mode inputs. +const FUEL_CP0_CONFIG: u32 = 0x6c9a_a985; +const FUEL_BOARD_ID: u64 = 0x4004; + +/// The local node register blocks projected into the processor's I/O view. +pub mod reg { + pub const PI1_BASE: u64 = 0x20_0000; + pub const IO_BASE: u64 = 0x40_0000; + pub const LB_BASE: u64 = 0x60_0000; + pub const NI_BASE: u64 = 0x68_0000; + pub const XB_BASE: u64 = 0x70_0000; + pub const MD_BASE: u64 = 0x78_0000; + pub const MD_SIZE: u64 = 0x8_0000; +} + + +/// Build the Fuel's processor, `part` being the one the config resolved, and +/// set the PROM boot-mode inputs. +pub(crate) fn build_cpu(bus: Arc, part: &Part) -> Box { + let cfg = MipsCpuConfig::for_model::(Arc::new(crate::ip35::memory_map())) + .with_prid(part.prid()); + let mut cpu = Box::new(MipsExecutor::new( + bus as Arc, + MipsTlb::new(cfg.tlb_entries), + &cfg, + )); + cpu.phys_mask = CPU_PHYS_MASK; + // Fuel's 500 MHz/2 MiB flash record selects these boot-mode inputs. + // A mismatch enters the PROM's error routine before GP is initialized. + cpu.core.cp0_config = FUEL_CP0_CONFIG; + cpu +} + +/// Shared device composition for a single-processor Fuel. +/// Graphics is absent; the serial workstation shares these base I/O devices. +pub(crate) struct FuelBoard { + pub bus: Arc, + /// Cacheable memory at plain physical addresses: what the cache fills + /// from, and the one bus the JIT and its compile pool read code through. + cached: Arc, + pub pi: [Arc; 2], + pub md: Arc, + pub l1: Arc, + pub flash: Arc, + pub lb: Arc, + pub io: Arc, + pub ni: Arc, + pub xb: Arc, + pub xbridge: Arc, + pub ioc3: Arc, + pub eth: Arc, + pub scsi: Arc, + pub usb: Arc, + /// The VPro board in the graphics slot, widget 13. + pub vpro: Option>, + /// The status bar's activity bits and the CPU's Compare matches, which + /// the graphics board shows. + pub heartbeat: Arc, + pub fasttick: Arc, + /// What the graphics board's status bar shows: the activity bits, the + /// Compare matches and the CPU's cycle count and Count rate. + pub status: Arc, + pub serial: [Arc; 3], + #[cfg(test)] + pub irq: Arc, + clock: Clock, + last_clock: AtomicU64, + running: AtomicBool, +} + +pub(crate) struct FuelBoardConfig<'a> { + pub payload: &'a [u8], + pub clock: Clock, + pub flash_path: Option<&'a Path>, + pub station_mac: [u8; 6], + pub timekeeper: Arc, + /// The IOC3's PS/2 keyboard and mouse controller. + pub ps2_input: bool, + /// A USB keyboard and mouse on the OHCI root ports. + pub usb_input: bool, + /// Replacements for the L1's reported identity; unset keeps its defaults. + pub l1: L1Identity, + /// `None`, `V12` or `V10`; validation refuses the others. + pub graphics: GraphicsBoard, + /// The VPro's fragment pipeline: worker count and tile size (`[graphics]`). + pub graphics_pool: (usize, u32), + /// Which unit draws and, for the GPU, the wgpu backend (`[graphics]`). + pub graphics_raster: (crate::config::RasterUnit, &'a str), +} + +/// What a Fuel config may replace in the L1's identity replies. +#[derive(Clone, Debug, Default)] +pub(crate) struct L1Identity { + pub revision: Option<[u32; 3]>, + pub brick_serial: Option, +} + +impl FuelBoard { + /// Flush the disks on both channels of the storage controller. + pub fn flush_disks(&self) -> std::io::Result<()> { + (0..2).try_for_each(|channel| self.scsi.bus_for_channel(channel).unwrap().flush()) + } + + #[cfg(test)] + pub fn new(payload: &[u8], clock: Clock) -> Result { + Self::with_config(FuelBoardConfig { + payload, clock, flash_path: None, + station_mac: L1Config::default().mac, + timekeeper: Arc::new(Timekeeper::fuel()?), + ps2_input: true, + usb_input: false, + l1: L1Identity::default(), + graphics: GraphicsBoard::None, + graphics_pool: (2, 32), + graphics_raster: (crate::config::RasterUnit::Cpu, "auto"), + }) + } + + pub fn with_config(config: FuelBoardConfig<'_>) -> Result { + let FuelBoardConfig { payload, clock, flash_path, station_mac, timekeeper, ps2_input, usb_input, l1: l1_identity, graphics, graphics_pool, graphics_raster } = config; + let md = BedrockMd::with_clock(clock.clone()); + let phys = md.physical().clone(); + let pi: [Arc; 2] = std::array::from_fn(|interface| { + BedrockPi::new( + BedrockPiConfig { + interface: interface as u8, + request_region: 0, + }, + clock.clone(), + ) + }); + // Only CPU A is fitted on the single-processor board. + pi[0].write64(pi_reg::CPU_A_ENABLE, 1); + let mut l1_config = L1Config { mac: station_mac, ..Default::default() }; + if let Some(revision) = l1_identity.revision { + l1_config.revision = Some(revision); + } + if let Some(serial) = l1_identity.brick_serial { + l1_config.brick_serial = Some(serial); + } + let l1 = Arc::new(L1::new(l1_config)?); + let flash = Arc::new(match flash_path { + Some(path) => Flash::with_file(payload, path), + None => Flash::new(payload), + }.map_err(|e| e.to_string())?); + let lb = Arc::new(BedrockLb::new(clock.clone())); + let io = Arc::new(BedrockIo::new()); + let ni = Arc::new(BedrockNi::new()); + let xb = Arc::new(BedrockXb::new()); + let mut registers = SparseWindowMap::default(); + registers.map( + 0, + PI_SIZE, + pi[0].clone(), + ); + registers.map( + reg::PI1_BASE, + PI_SIZE, + pi[1].clone(), + ); + registers.map( + reg::IO_BASE, + IO_SIZE, + io.clone(), + ); + registers.map( + reg::LB_BASE, + LB_SIZE, + lb.clone(), + ); + registers.map( + reg::NI_BASE, + NI_SIZE, + ni.clone(), + ); + registers.map( + reg::XB_BASE, + XB_SIZE, + xb.clone(), + ); + registers.map( + reg::MD_BASE, + reg::MD_SIZE, + md.md_bus(), + ); + let registers = Arc::new(registers); + io.connect_hub(Arc::new(IoHub(pi.clone()))); + io.connect_memory(md.dma_view()); + let p = pi.clone(); + lb.connect_error(Arc::new(move |level| { + for pi in &p { + pi.set_lb_error(level); + } + })); + let p = pi.clone(); + ni.connect_error(Arc::new(move |level| { + for pi in &p { + pi.set_ni_error(level); + } + })); + let p = pi.clone(); + xb.connect_error(Arc::new(move |level| { + for pi in &p { + pi.set_xb_error(level); + } + })); + let io_node = Arc::downgrade(&io); + lb.connect_node_changed(Arc::new(move |node| { + if let Some(io) = io_node.upgrade() { + io.set_node_number(node); + } + })); + let p = pi.clone(); + md.on_error(Arc::new(move |kind| { + for pi in &p { + match kind { + MemoryError::Correctable => pi.set_pending(1, 61), + MemoryError::Uncorrectable => { + pi.set_memory_uncorrectable(); + } + } + } + })); + #[cfg(test)] + let irq = Arc::new(PendingIrq::default()); + #[cfg(test)] + pi[0].set_interrupt_controller(irq.clone()); + l1.set_irq(IrqLine::new(Arc::new(UartIrq(pi.clone())), 0)); + let serial = std::array::from_fn(|_| Arc::new(CiSerialBackend::new())); + l1.set_serial_backend(serial[0].clone()); + let xbridge = XBridge::new(md.dma_view()); + for widget in [0, 10, 14, 15] { + io.connect_widget(widget, xbridge.widget(widget).unwrap()); + } + let weak_io = Arc::downgrade(&io); + xbridge.set_interrupt_sender(Arc::new(move |packet| { + if packet.widget != 10 { + return false; + } + weak_io.upgrade().is_some_and(|io| { + io.inbound_write( + packet.source_widget, + None, + packet.address, + 8, + packet.vector as u64 | if packet.level { 1 << 8 } else { 0 }, + ) == BUS_OK + }) + })); + let weak_io = Arc::downgrade(&io); + xbridge.set_dma_sender(Arc::new(move |packet, bytes| { + if packet.widget != 10 || packet.half_read_requested || packet.half_write_requested { + dlog!(LogModule::Bedrock, + "Fuel: unsupported XBridge DMA destination or halfword placement {packet:?}" + ); + return false; + } + let Some(io) = weak_io.upgrade() else { + return false; + }; + // Preserve the width of XBridge's naturally aligned packet legs, + // including Hub registers whose narrower accesses are refused. + let size = bytes.len(); + if matches!(size, 1 | 2 | 4 | 8) + && packet.address & (size as u64 - 1) == 0 + { + if packet.write { + let value = bytes.iter().fold(0u64, |value, byte| (value << 8) | *byte as u64); + return io.inbound_write( + packet.source_widget, Some(packet.slot), packet.address, size as u8, value, + ) == BUS_OK; + } + let value = io.inbound_read( + packet.source_widget, Some(packet.slot), packet.address, size as u8, + ); + if value.status != BUS_OK { + return false; + } + bytes.copy_from_slice(&value.data.to_be_bytes()[8 - size..]); + return true; + } + for (i, byte) in bytes.iter_mut().enumerate() { + let a = packet.address + i as u64; + if packet.write { + if io.inbound_write(packet.source_widget, Some(packet.slot), a, 1, *byte as u64) + != BUS_OK + { + return false; + } + } else { + let value = io.inbound_read(packet.source_widget, Some(packet.slot), a, 1); + if value.status != BUS_OK { + return false; + } + *byte = value.data as u8; + } + } + true + })); + let mac = l1.station_mac(); + let mut ioc3_config = Ioc3Config::fuel(mac); + ioc3_config.keyboard_mouse = ps2_input; + let ioc3 = Arc::new(Ioc3::new(ioc3_config)); + let eth = Ioc3Eth::new_fuel(mac); + ioc3.attach_eth(eth.clone()); + ioc3.attach_timekeeper(timekeeper); + ioc3.set_serial_backend(0, serial[1].clone()); + ioc3.set_serial_backend(1, serial[2].clone()); + let scsi = Arc::new(Isp12160::new( + Isp12160::new_bus(), + Isp12160::new_bus(), + )); + let heartbeat = Arc::new(AtomicU64::new(0)); + let fasttick = Arc::new(AtomicU64::new(0)); + scsi.set_heartbeat(heartbeat.clone()); + ioc3.set_heartbeat(heartbeat.clone()); + eth.set_heartbeat(heartbeat.clone()); + let status = crate::disp::StatusSource::new(heartbeat.clone(), fasttick.clone()); + let vpro = if graphics == GraphicsBoard::None { + None + } else { + // Interrupt packets leave through the crossbar as the heads' do; + // DMA enters Bedrock as this widget's inbound traffic. + let crossbar = Arc::downgrade(&xbridge); + let packets: PacketSender = Arc::new(move |widget, address, vector, level| { + crossbar.upgrade().is_some_and(|x| x.send_interrupt(InterruptPacket { + source_widget: WIDGET_VPRO, + widget, + address, + vector, + level, + })) + }); + let dma = Arc::new(WidgetDma { io: Arc::downgrade(&io), widget: WIDGET_VPRO }); + let board = Odyssey::new_widget_relative( + WIDGET_VPRO, graphics.as_str(), packets, dma, status.clone(), graphics_pool, graphics_raster, + )?; + xbridge.attach_widget(WIDGET_VPRO, board.clone()); + io.connect_widget(WIDGET_VPRO, xbridge.widget(WIDGET_VPRO).expect("attached widget")); + Some(board) + }; + // Each processor interface keeps its own graphics registers. + for pi in &pi { + pi.set_node_hooks(Arc::new(crate::dev::bedrock_pi::GraphicsRegisters::default())); + } + let pci = xbridge.bus(15); + let slot = |device| DevFn { device, function: 0 }; + pci.attach(slot(1), scsi.clone()); + pci.attach(slot(4), ioc3.clone()); + let usb = Ohci::new(); + usb.set_clock(clock.clone()); + if usb_input { + // Host input reaches the USB devices through the same controller + // object the GUI, VNC and CI already drive; each device drops + // what its guest has not enabled. + let keyboard = UsbKeyboard::new(); + let mouse = UsbMouse::new(); + usb.attach(0, keyboard.clone())?; + usb.attach(1, mouse.clone())?; + ioc3.ps2().add_sink(keyboard); + ioc3.ps2().add_sink(mouse); + } + pci.attach(slot(5), usb.clone()); + ioc3.connect_superio_interrupt(pci.board_irq(BoardIrq(0))); + let reset_ioc3 = ioc3.clone(); + let reset_eth = eth.clone(); + let reset_scsi = scsi.clone(); + let reset_usb = usb.clone(); + let reset_vpro = vpro.as_ref().map(Arc::downgrade); + xbridge.set_reset_output(Arc::new(move |port| { + if port == 15 { + reset_ioc3.power_on(); + reset_eth.power_on(); + reset_scsi.power_on(); + reset_usb.power_on(); + } + // A link reset of the graphics port resets the board behind it. + if port == WIDGET_VPRO { + if let Some(vpro) = reset_vpro.as_ref().and_then(|v| v.upgrade()) { + vpro.power_on(); + } + } + })); + let mut bus = Ip35Bus::new(phys.clone()); + md.rebuild_memory(); + bus.set_fallback(1, 1 << 32, Arc::new(IoPio(io.clone()))); + // The two XIO register forms select the requester's PI0 interface. + let permission_md = Arc::downgrade(&md); + let permission: Arc bool + Send + Sync> = Arc::new(move || { + permission_md.upgrade().is_some_and(|md| md.current_cell() & 1 != 0) + }); + let protected_regs = Arc::new(Window::new(0, 0x80_0000, registers.clone()).permitted(permission.clone())); + for base in [0x0100_0000, 0x0180_0000] { + bus.map(1, base, 0x80_0000, protected_regs.clone()); + } + // Cached RAM uses MD's direct bank mappings; diagnostic views stay separate. + bus.map(3, 0, 1 << 33, md.uncached_view()); + // Hub-special low memory: its first 256 MiB is the node memory view. + bus.map(0, 0, 0x1000_0000, md.uncached_view()); + let prom = Arc::new( + flash + .array_view(0, PART_SIZE) + .map_err(|e| e.to_string())?, + ); + let boot = Arc::new(flash.boot_view()); + for base in [0x1800_0000, 0x3800_0000] { + bus.map(0, base, 0x800_0000, boot.clone()); + } + // Reset execution always sees stored bytes, independently of commands. + bus.map(0, 0x1fc0_0000, 0x40_0000, prom.clone()); + if !payload.is_empty() { + map_device( + &phys, + 0x1fc0_0000, + payload.len() as u64, + prom.clone(), + "Fuel cached reset PROM", + ); + } + let leds = Arc::new(Leds { + md: md.clone(), + values: Mutex::new([0; 4]), + }); + // The three windows occupy the same processor register page. + let special = Arc::new(HubSpecialPage { + windows: [ + Window::new(0, 0x80, Arc::new(HubHead)).permitted(permission.clone()), + Window::new(0x80, 0x40, l1.clone()).permitted(permission.clone()), + Window::new(0xc0, 0x20, leds).permitted(permission.clone()), + ], + }); + for base in [0x1000_0000, 0x3000_0000] { + bus.map(0, base, 1 << SLOT_SHIFT, special.clone()); + bus.map(0, base + 0x400_0000, 8, Arc::new(BoardId(md.clone()))); + } + bus.map(0, 0x1_c000_0000, 1 << 30, md.directory_bus()); + bus.map(0, 0x1_4000_0000, 1 << 30, md.ecc_bus()); + bus.map(0, 0x1_0000_0000, 1 << 30, md.ecc_scrub_bus()); + bus.map(2, 0, 1 << 33, md.fetchop_bus()); + let bus = Arc::new(bus); + let cached = bus.cached_view(md.dma_view()); + Ok(Self { + bus, + cached, + pi, + md, + l1, + flash, + lb, + io, + ni, + xb, + xbridge, + ioc3, + eth, + scsi, + usb, + vpro, + heartbeat, + fasttick, + status, + serial, + #[cfg(test)] + irq, + last_clock: AtomicU64::new(clock()), + clock, + running: AtomicBool::new(false), + }) + } + + pub fn cached_bus(&self) -> Arc { + self.cached.clone() + } + + pub fn build_cpu(&self, part: &Part) -> Box { + let mut cpu = build_cpu(self.bus.clone(), part); + cpu.cache = R12000ShadowCache::new(self.cached_bus()); + cpu.set_cached_bus(self.cached_bus()); + cpu + } + + pub fn start(&self) { + self.last_clock + .store((self.clock)(), Ordering::Release); + self.md.start_clock(); + self.running + .store(true, Ordering::Release); + for pi in &self.pi { + pi.start(); + } + self.l1.start(); + self.ioc3.start(); + self.eth.start(); + self.scsi.start(); + self.usb.start(); + if let Some(vpro) = &self.vpro { + vpro.start(); + } + } + /// The board's devices that take monitor commands: the physical map, the + /// L1, the SCSI controller, the keyboard and mouse (`ps2 type` and + /// friends) and, when one is fitted, the VPro (`vpro …`). + pub fn monitor_devices(&self) -> Vec> { + let mut devices: Vec> = vec![ + self.md.physical().clone(), + self.l1.clone(), + self.scsi.clone(), + self.ioc3.ps2(), + ]; + if let Some(vpro) = &self.vpro { + devices.push(vpro.clone()); + } + devices + } + + pub fn poll(&self) { + if self.running.load(Ordering::Acquire) { + let now = (self.clock)(); + let previous = self + .last_clock + .swap(now, Ordering::AcqRel); + self.flash.advance_ns(now.saturating_sub(previous)); + self.usb.sync_clock(); + for pi in &self.pi { + pi.poll(); + } + } + } + pub fn stop(&self) { + self.poll(); + self.running + .store(false, Ordering::Release); + self.md.stop_clock(); + if let Some(vpro) = &self.vpro { + vpro.stop(); + } + self.usb.stop(); + self.scsi.stop(); + self.eth.stop(); + self.ioc3.stop(); + self.l1.stop(); + for pi in &self.pi { + pi.stop(); + } + } +} + +/// The crossbar widget of the Fuel's graphics slot: a Fuel IRIX inventory +/// lists its VPro under xtalk/13, and OpenBSD finds Odyssey at widget 13. +pub const WIDGET_VPRO: u8 = 13; + +/// A widget's DMA as Bedrock's inbound path takes it: crosstalk addresses, +/// with the widget's inbound permissions applied. +struct WidgetDma { + io: std::sync::Weak, + widget: u8, +} +macro_rules! widget_dma_access { + ($r:ident,$w:ident,$ty:ty,$result:ident,$size:expr) => { + fn $r(&self, a: u64) -> $result { + let Some(io) = self.io.upgrade() else { return $result::err() }; + let r = io.inbound_read(self.widget, None, a, $size); + $result { data: r.data as $ty, status: r.status } + } + fn $w(&self, a: u64, v: $ty) -> u32 { + self.io.upgrade().map_or(BUS_ERR, |io| io.inbound_write(self.widget, None, a, $size, v as u64)) + } + }; +} +impl BusDevice for WidgetDma { + widget_dma_access!(read8, write8, u8, BusRead8, 1); + widget_dma_access!(read16, write16, u16, BusRead16, 2); + widget_dma_access!(read32, write32, u32, BusRead32, 4); + widget_dma_access!(read64, write64, u64, BusRead64, 8); +} + +// XIO register packets reach only the two PI interrupt-modify registers. +// The processor's register projection has a separate, wider interface. +struct IoHub([Arc; 2]); +impl BusDevice for IoHub { + fn write64(&self, a: u64, v: u64) -> u32 { + if matches!(a, 0x90 | 0x20_0090) && self.0[(a >> 21) as usize].io_modify(v, 0) { + BUS_OK + } else { + BUS_ERR + } + } +} + +struct HubSpecialPage { + windows: [Window; 3], +} +impl HubSpecialPage { + fn target(&self, a: u64, n: u64) -> Option<&Window> { + self.windows.iter().find(|window| window.contains(a, n)) + } +} +macro_rules! hub_page_access { + ($r:ident,$w:ident,$t:ty,$out:ident,$n:expr) => { + fn $r(&self, a: u64) -> $out { + self.target(a, $n).map(|window| window.$r(a)).unwrap_or_else($out::err) + } + fn $w(&self, a: u64, value: $t) -> u32 { + self.target(a, $n).map(|window| window.$w(a, value)).unwrap_or(BUS_ERR) + } + }; +} +impl BusDevice for HubSpecialPage { + hub_page_access!(read8, write8, u8, BusRead8, 1); + hub_page_access!(read16, write16, u16, BusRead16, 2); + hub_page_access!(read32, write32, u32, BusRead32, 4); + hub_page_access!(read64, write64, u64, BusRead64, 8); +} + +struct BoardId(Arc); +impl BusDevice for BoardId { + fn read64(&self, a: u64) -> BusRead64 { + if a == 0 && self.0.current_cell() & 1 != 0 { + BusRead64::ok(FUEL_BOARD_ID) + } else { + BusRead64::err() + } + } +} +// Compatibility head windows: their register contract is not established. +// PROM 6.210 writes 0x2000 at offsets 0 and 0x40 before DRAM discovery. +// Retain the existing zero-read/acknowledged-write model, with accesses +// observable through the Bedrock log, until the register behavior is known. +struct HubHead; +macro_rules! head_access { + ($r:ident,$w:ident,$ty:ty,$result:ident) => { + fn $r(&self, a: u64) -> $result { + dlog!(LogModule::Bedrock, + "Fuel: unidentified Hub-special read {} bytes at {a:#x}", + std::mem::size_of::<$ty>() + ); + $result::ok(0) + } + fn $w(&self, a: u64, v: $ty) -> u32 { + dlog!(LogModule::Bedrock, + "Fuel: unidentified Hub-special write {} bytes at {a:#x} value {v:#x}", + std::mem::size_of::<$ty>() + ); + BUS_OK + } + }; +} +impl BusDevice for HubHead { + head_access!(read8, write8, u8, BusRead8); + head_access!(read16, write16, u16, BusRead16); + head_access!(read32, write32, u32, BusRead32); + head_access!(read64, write64, u64, BusRead64); +} +struct Leds { + md: Arc, + values: Mutex<[u8; 4]>, +} +impl BusDevice for Leds { + fn read64(&self, a: u64) -> BusRead64 { + if a < 32 && a & 7 == 0 && self.md.current_cell() & 1 != 0 { + BusRead64::ok(self.values.lock()[a as usize / 8] as u64) + } else { + BusRead64::err() + } + } + fn write64(&self, a: u64, v: u64) -> u32 { + if a < 32 && a & 7 == 0 && self.md.current_cell() & 1 != 0 { + self.values.lock()[a as usize / 8] = v as u8; + BUS_OK + } else { + BUS_ERR + } + } +} + +struct IoPio(Arc); +macro_rules! pio_access { + ($r:ident,$w:ident,$ty:ty,$result:ident,$size:expr) => { + fn $r(&self, a: u64) -> $result { + let r = self.0.pio_read(a, $size); + $result { + data: r.data as $ty, + status: r.status, + } + } + fn $w(&self, a: u64, v: $ty) -> u32 { + self.0.pio_write(a, $size, v as u64) + } + }; +} +impl BusDevice for IoPio { + pio_access!(read8, write8, u8, BusRead8, 1); + pio_access!(read16, write16, u16, BusRead16, 2); + pio_access!(read32, write32, u32, BusRead32, 4); + pio_access!(read64, write64, u64, BusRead64, 8); +} +impl Drop for FuelBoard { + fn drop(&mut self) { + self.stop(); + } +} + +#[cfg(test)] +#[derive(Default)] +pub(crate) struct PendingIrq(AtomicU64); +#[cfg(test)] +impl PendingIrq { + pub fn bits(&self) -> u64 { + self.0.load(Ordering::Acquire) + } +} +#[cfg(test)] +impl InterruptController for PendingIrq { + fn set_line(&self, line: u32, level: bool) { + if line >= 5 { + return; + } + let mask = 1 << (10 + line); + if level { + self.0.fetch_or(mask, Ordering::Release); + } else { + self.0 + .fetch_and(!mask, Ordering::Release); + } + } +} +struct UartIrq([Arc; 2]); +impl InterruptController for UartIrq { + fn set_line(&self, _: u32, level: bool) { + for pi in &self.0 { + pi.set_uart_irq(level); + } + } +} diff --git a/src/lib.rs b/src/lib.rs index 48915580d..2c809e675 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -190,6 +190,7 @@ pub mod ip27_onyx2; pub mod ip27_origin2000; /// Fuel has an IP34 board and uses the IP35 software platform and processor views. pub mod ip35; +pub mod ip35_machine; pub mod machine_state; pub mod platform; pub mod physical; diff --git a/src/machine.rs b/src/machine.rs index 9efbfe6e6..d343154b1 100644 --- a/src/machine.rs +++ b/src/machine.rs @@ -53,7 +53,7 @@ pub fn set_emulator_profile(profile: MachineProfile) { fn emulator_profile() -> Option { let raw = EMULATOR_PROFILE.load(std::sync::atomic::Ordering::Relaxed); - [MachineProfile::IndyIp24, MachineProfile::Indigo2Ip22, MachineProfile::Indigo2Ip28, MachineProfile::O2Ip32, MachineProfile::OctaneIp30, MachineProfile::Origin200Ip29, MachineProfile::Origin2000Ip27, MachineProfile::Onyx2Ip27] + [MachineProfile::IndyIp24, MachineProfile::Indigo2Ip22, MachineProfile::Indigo2Ip28, MachineProfile::O2Ip32, MachineProfile::OctaneIp30, MachineProfile::Origin200Ip29, MachineProfile::Origin2000Ip27, MachineProfile::Onyx2Ip27, MachineProfile::FuelIp34] .into_iter().find(|p| *p as u8 == raw) } @@ -71,6 +71,7 @@ fn emulator_name_thirds(profile: Option) -> &'static [&'static s Some(MachineProfile::Origin200Ip29) => &["IRIX", "Iris", "IPsomething", "IP29"], Some(MachineProfile::Origin2000Ip27) => &["IRIX", "Iris", "IPsomething", "IP27"], Some(MachineProfile::Onyx2Ip27) => &["IRIX", "Iris", "IPsomething", "IP27"], + Some(MachineProfile::FuelIp34) => &["IRIX", "Iris", "IPsomething", "IP34"], None => &["IRIX", "Iris", "IPsomething"], } } @@ -800,33 +801,22 @@ impl Machine { mc.set_cpu(Arc::downgrade(&cpu_device)); ioc.set_interrupts(cpu.interrupts_ptr()); - // ppmem: hand the CPU's inline bitmap field to the memory space, so a - // MEMCFG remap publishes straight into the object the CPU executes out - // of. Safe per `set_bitmap_sink`: the executor is at its final address - // by now and the only writer is this same CPU thread's MEMCFG path. + // ppmem: the CPU's cache and the JIT read RAM through a cached view of + // the window that is the CPU's own, so what the cache holds stays out + // of every other master's sight (`PpMemSpace::add_cached_view`). Its + // bitmap is published straight into the CPU's inline fields, the + // executor's and the cache's, which are at their final addresses by + // now; the only writer is this CPU thread's MEMCFG path. A cache with + // no bitmap field (the full cache without tcache) returns null, which + // the view skips. Physical's direct paths keep the shared window. { use crate::ppmem::MappedMemory; if let Some(sp) = phys.ppmem_space() { - unsafe { sp.set_bitmap_sink(cpu.ppmem_bitmap_ptr()) }; - } - // Point Physical's cached bitmap at the CPU's field too, or its - // direct-access fast path would keep reading the sink the space - // just abandoned. - phys.resync_ppmem_bitmap(); - // The cache reads RAM straight out of the same window: the full - // cache under tcache, the shadow cache always. A cache that does - // not (the full cache without tcache) has no bitmap field and - // returns null. Must come after set_bitmap_sink, so it captures - // the CPU's inline field rather than the sink the space just - // abandoned. - if let Some(sp) = phys.ppmem_space() { - unsafe { cpu.set_tcache_window(sp.window_base()) }; - // The cache keeps the bitmap inline; ppmem publishes into it - // directly on every remap, so the hot path needs no indirection. - let bitmap = cpu.tcache_bitmap_ptr(); - if !bitmap.is_null() { - unsafe { sp.set_bitmap_sink2(bitmap) }; - } + let v = sp + .add_cached_view(&[cpu.ppmem_bitmap_ptr(), cpu.tcache_bitmap_ptr()]) + .expect("ppmem: the CPU's cached view"); + unsafe { cpu.set_tcache_window(sp.view_base(v)) }; + unsafe { cpu.set_cached_view(sp, v) }; // Window writes bypass BusDevice, so the cache needs the gen // window to bump page counters itself — otherwise jitv2 keeps // running compiled code for pages the guest has modified. @@ -2177,11 +2167,12 @@ mod emulator_name_tests { Some(MachineProfile::Origin200Ip29) => &["Indy", "Indigo2", "IP24", "IP22", "IP28", "IP30", "IP32", "O2"], Some(MachineProfile::Origin2000Ip27) => &["Indy", "Indigo2", "IP24", "IP22", "IP28", "IP30", "IP32", "O2"], Some(MachineProfile::Onyx2Ip27) => &["Indy", "Indigo2", "IP24", "IP22", "IP28", "IP30", "IP32", "O2"], + Some(MachineProfile::FuelIp34) => &["Indy", "Indigo2", "IP24", "IP22", "IP28", "IP30", "IP32", "O2"], None => &["Indy", "Indigo2", "IP24", "IP22", "IP28", "IP30", "IP32", "O2"], } }; for p in [None, Some(MachineProfile::IndyIp24), Some(MachineProfile::Indigo2Ip22), - Some(MachineProfile::Indigo2Ip28), Some(MachineProfile::O2Ip32), Some(MachineProfile::OctaneIp30), Some(MachineProfile::Origin200Ip29), Some(MachineProfile::Origin2000Ip27), Some(MachineProfile::Onyx2Ip27)] { + Some(MachineProfile::Indigo2Ip28), Some(MachineProfile::O2Ip32), Some(MachineProfile::OctaneIp30), Some(MachineProfile::Origin200Ip29), Some(MachineProfile::Origin2000Ip27), Some(MachineProfile::Onyx2Ip27), Some(MachineProfile::FuelIp34)] { let thirds = emulator_name_thirds(p); assert!(!thirds.is_empty()); for w in thirds { @@ -2193,7 +2184,7 @@ mod emulator_name_tests { #[test] fn profile_round_trips_through_the_global() { - for p in [MachineProfile::IndyIp24, MachineProfile::Indigo2Ip22, MachineProfile::Indigo2Ip28, MachineProfile::O2Ip32, MachineProfile::OctaneIp30, MachineProfile::Origin200Ip29, MachineProfile::Origin2000Ip27, MachineProfile::Onyx2Ip27] { + for p in [MachineProfile::IndyIp24, MachineProfile::Indigo2Ip22, MachineProfile::Indigo2Ip28, MachineProfile::O2Ip32, MachineProfile::OctaneIp30, MachineProfile::Origin200Ip29, MachineProfile::Origin2000Ip27, MachineProfile::Onyx2Ip27, MachineProfile::FuelIp34] { set_emulator_profile(p); assert_eq!(emulator_profile(), Some(p)); } diff --git a/src/pci.rs b/src/pci.rs index 679e23fe8..7a8651e91 100644 --- a/src/pci.rs +++ b/src/pci.rs @@ -1,814 +1,26 @@ //! PCI: the configuration header every function has, and the trait a //! function implements to sit behind a host bridge. //! -//! The bridge (the root complex) owns everything machine-specific: how -//! configuration cycles are generated, where the CPU sees PCI memory, byte -//! lanes, and which interrupt input each slot's pins are wired to. A -//! function only sees configuration registers, accesses to its BARs, and an -//! [`IrqLine`] for its pin. - -use std::sync::{Arc, Mutex}; - -use crate::irq::IrqLine; -use crate::snapshot::{get_field, load_u32_slice, u32_slice_to_toml}; -use crate::traits::{BusDevice, BusRead16, BusRead32, BusRead64, BusRead8, BUS_OK}; - -/// Command register bits. -pub const PCI_COMMAND_IO: u32 = 0x1; -pub const PCI_COMMAND_MEMORY: u32 = 0x2; - -/// Interrupt pins, as register 0x3d holds them. -pub const PCI_INTA: u8 = 1; - -/// What one base address register decodes, and how much. -#[derive(Clone, Copy, PartialEq, Eq, Debug)] -pub enum Bar { - Unused, - /// I/O space, this many bytes (a power of two). - Io(u32), - /// 32-bit memory space, this many bytes (a power of two). - Mem(u32), -} - -impl Bar { - fn size(self) -> u32 { - match self { - Bar::Unused => 0, - Bar::Io(n) | Bar::Mem(n) => n, - } - } -} - -/// Which address space an access is in. -#[derive(Clone, Copy, PartialEq, Eq, Debug)] -pub enum Space { - Io, - Mem, -} - -/// Initial value and writable bits for a function-specific configuration dword. -#[derive(Clone, Copy, Debug)] -pub struct PciRegisterPolicy { - /// Dword index. Header, BAR, subsystem identity, and interrupt dwords are managed separately. - pub reg_dword: usize, - /// Value restored by [`PciConfig::reset`]. - pub initial: u32, - /// Bits that configuration writes may change. - pub writable_mask: u32, -} - -/// Function-specific write behavior for a type 0 PCI configuration header. -/// -/// Register policies are copied into the header during construction; the -/// borrowed slice is not retained. -#[derive(Clone, Copy, Debug)] -pub struct PciConfigPolicy<'a> { - /// Fixed Status bits that read as one. - pub status: u16, - /// Device-latched Status bits that writes acknowledge with one. - pub status_w1c_mask: u16, - /// Writable Command bits. - pub command_mask: u16, - /// Writable bits in the latency timer byte. - pub latency_mask: u8, - /// Writable bits in the cache-line-size byte. - pub cache_line_mask: u8, - /// Initial values and write masks for function-specific dwords. - pub registers: &'a [PciRegisterPolicy], -} - -impl<'a> Default for PciConfigPolicy<'a> { - fn default() -> Self { - Self { - status: 0x0200, - status_w1c_mask: 0, - command_mask: 0xffff, - latency_mask: 0xff, - cache_line_mask: 0xff, - registers: &[], - } - } -} - -/// One function's type 0 configuration header. -/// -/// Enough of it to be discovered and configured: identity, class, command -/// and status, the interrupt pin, and base address registers that answer a -/// size probe. Writing all ones to a BAR and reading it back is how firmware -/// learns how much space a device wants, so a BAR that stores whatever it is -/// given reports a size of four gigabytes and the bus layout falls apart. -pub struct PciConfig { - regs: Mutex<[u32; 64]>, - bars: [Bar; 6], - /// The registers as the function powers up, for [`PciConfig::reset`]. - power_on: [u32; 64], - /// Dwords `16..caps_end` hold a capability list, which firmware cannot - /// overwrite. 0 when the function has none. - caps_end: usize, - status: u16, - status_w1c_mask: u16, - command_mask: u16, - latency_mask: u8, - cache_line_mask: u8, - writable_masks: [u32; 64], -} - -impl PciConfig { - /// `class` is the 24-bit class/subclass/interface, as it sits in the top - /// three bytes of register 0x08. `pin` is 0 for none, or [`PCI_INTA`] - /// and on; the line register is left for firmware to assign. - pub fn new(vendor: u16, device: u16, class: u32, revision: u8, bars: [Bar; 6], pin: u8) -> Self { - Self::new_with_policy( - vendor, - device, - class, - revision, - bars, - pin, - PciConfigPolicy::default(), - ) - } - - /// Construct a type 0 header with function-specific register behavior. - pub fn new_with_policy( - vendor: u16, - device: u16, - class: u32, - revision: u8, - bars: [Bar; 6], - pin: u8, - policy: PciConfigPolicy<'_>, - ) -> Self { - let mut r = [0u32; 64]; - r[0] = ((device as u32) << 16) | vendor as u32; - r[2] = (class << 8) | revision as u32; - r[15] = (pin as u32) << 8; - for (i, b) in bars.iter().enumerate() { - if let Bar::Io(_) = b { - r[4 + i] = 1; - } - } - let mut writable_masks = [u32::MAX; 64]; - for register in policy.registers { - assert!( - matches!(register.reg_dword, 10 | 12..=14 | 16..=63), - "pci: register policy requires a function-specific dword in 0..64"); - r[register.reg_dword] = register.initial; - writable_masks[register.reg_dword] = register.writable_mask; - } - Self { - regs: Mutex::new(r), - bars, - power_on: r, - status: policy.status, - status_w1c_mask: policy.status_w1c_mask, - command_mask: policy.command_mask, - latency_mask: policy.latency_mask, - cache_line_mask: policy.cache_line_mask, - writable_masks, - caps_end: 0, - } - } - - /// Give the function a capability list: `caps` are the raw dwords of the - /// list, which starts at dword `first` (at least 16, the end of the - /// header). They are read-only and survive a reset. The header's - /// capability pointer and status bit are set to match. - pub fn with_capabilities(mut self, first: usize, caps: &[u32]) -> Self { - assert!(first >= 16 && first + caps.len() <= 64, "capabilities must fit in the 256-byte header"); - let r = self.regs.get_mut().unwrap(); - for (i, &d) in caps.iter().enumerate() { - r[first + i] = d; - } - // Capabilities pointer (byte offset) in the low byte of dword 13. - r[13] = (first * 4) as u32; - self.power_on = *r; - self.caps_end = first + caps.len(); - self - } - - /// RST#: back to the power-up header. The command register clears, so - /// the function stops decoding, and the BARs and line forget what - /// firmware assigned. - pub fn reset(&self) { - let mut r = self.regs.lock().unwrap(); - // The subsystem identity is the board's, set once after - // construction (IOC3's selects what a guest instantiates). - let subsystem = r[11]; - *r = self.power_on; - r[11] = subsystem; - } - - /// Register `reg` (a dword index, 0..64). - pub fn read(&self, reg: usize) -> u32 { - let r = self.regs.lock().unwrap(); - match reg { - // A function with a capability list also sets status bit 4. - 1 => (r[1] & (0xffff | ((self.status_w1c_mask as u32) << 16))) - | ((self.status as u32) << 16) - | if r[13] != 0 { 0x0010_0000 } else { 0 }, - _ => r[reg], - } - } - - pub fn write(&self, reg: usize, val: u32) { - self.write_masked(reg, val, u32::MAX); - } - - /// Write selected bits of dword `reg`. Both `val` and `mask` use the - /// register's bit positions. Unselected Status events are not acknowledged. - pub fn write_masked(&self, reg: usize, val: u32, mask: u32) { - if mask == 0 { - return; - } - let mut r = self.regs.lock().unwrap(); - match reg { - // Identity, class and header type are read-only. - 0 | 2 => {} - 3 => { - let mask = mask & (((self.latency_mask as u32) << 8) | self.cache_line_mask as u32); - r[3] = (r[3] & !mask) | (val & mask); - } - // Command writes replace selected bits; Status events clear on ones. - 1 => { - let clear = val & mask & ((self.status_w1c_mask as u32) << 16); - r[1] = (r[1] & !(mask & 0xffff) & !clear) - | (val & mask & self.command_mask as u32); - } - 4..=9 => { - let val = (r[reg] & !mask) | (val & mask); - let bar = self.bars[reg - 4]; - r[reg] = match bar { - Bar::Unused => 0, - // Address bits above the size take the write; those - // below read back as zero, which tells firmware the - // size. The type bits stay what the BAR is. - Bar::Io(n) => (val & !(n - 1)) | 1, - Bar::Mem(n) => val & !(n - 1), - }; - } - // The capability list is read-only. - 16.. if reg < self.caps_end => {} - // Interrupt pin is read-only; the line is firmware's scratch. - 15 => { - let mask = mask & !0xff00; - r[15] = (r[15] & !mask) | (val & mask); - } - _ => { - let mask = mask & self.writable_masks[reg]; - r[reg] = (r[reg] & !mask) | (val & mask); - } - } - } - - /// Latch selected device Status events without replacing earlier events. - pub fn set_status_bits(&self, bits: u16) { - self.regs.lock().unwrap()[1] |= ((bits & self.status_w1c_mask) as u32) << 16; - } - - /// Set the board-assigned Subsystem ID, which survives function reset. - pub fn set_subsystem_identity(&self, value: u32) { - self.regs.lock().unwrap()[11] = value; - } - - /// The base address firmware programmed into BAR `i`, type bits masked. - pub fn bar_base(&self, i: usize) -> u32 { - let v = self.regs.lock().unwrap()[4 + i]; - match self.bars[i] { - Bar::Io(_) => v & !3, - _ => v & !15, - } - } - - pub fn command(&self) -> u32 { - self.regs.lock().unwrap()[1] & 0xffff - } - - pub fn interrupt_pin(&self) -> u8 { - (self.regs.lock().unwrap()[15] >> 8) as u8 - } - - /// The windows this function decodes: (space, BAR, base, size) for - /// each BAR that has been assigned an address in a space the command - /// register enables, in BAR order. A bridge caches these and rebuilds - /// them after a configuration write. - pub fn windows(&self) -> Vec<(Space, usize, u32, u32)> { - let cmd = self.command(); - (0..6) - .filter_map(|i| { - let (space, size) = match self.bars[i] { - Bar::Io(n) if cmd & PCI_COMMAND_IO != 0 => (Space::Io, n), - Bar::Mem(n) if cmd & PCI_COMMAND_MEMORY != 0 => (Space::Mem, n), - _ => return None, - }; - let base = self.bar_base(i); - (base != 0).then_some((space, i, base, size)) - }) - .collect() - } - - /// The BAR decoding `addr` in `space`, and the offset within it: only - /// once the command register enables that space, and only for a BAR - /// that has been assigned an address. - pub fn decode(&self, space: Space, addr: u32) -> Option<(usize, u32)> { - self.decode_with(space, addr, false) - } - - /// [`Self::decode`], but a BAR left at zero decodes at zero: what a - /// function does on a bus whose firmware enables it without moving it - /// (the Octane's PROM reaches the RAD that way). - pub fn decode_unmoved(&self, space: Space, addr: u32) -> Option<(usize, u32)> { - self.decode_with(space, addr, true) - } - - fn decode_with(&self, space: Space, addr: u32, zero_ok: bool) -> Option<(usize, u32)> { - let enable = match space { - Space::Io => PCI_COMMAND_IO, - Space::Mem => PCI_COMMAND_MEMORY, - }; - if self.command() & enable == 0 { - return None; - } - (0..6).find_map(|i| { - let bar = self.bars[i]; - let in_space = matches!((bar, space), (Bar::Io(_), Space::Io) | (Bar::Mem(_), Space::Mem)); - let base = self.bar_base(i); - let size = bar.size(); - (in_space && (base != 0 || zero_ok) && addr.wrapping_sub(base) < size).then(|| (i, addr - base)) - }) - } -} - -/// The header's registers. Identity, class and the BARs' sizes come from -/// the function, so a header only loads into the same kind of function. -impl crate::traits::Saveable for PciConfig { - fn save_state(&self) -> toml::Value { - let mut t = toml::map::Map::new(); - t.insert("regs".into(), u32_slice_to_toml(&*self.regs.lock().unwrap())); - toml::Value::Table(t) - } - - fn load_state(&self, v: &toml::Value) -> Result<(), String> { - let Some(r) = get_field(v, "regs") else { return Ok(()) }; - let mut saved = [0u32; 64]; - load_u32_slice(r, &mut saved); - let mut regs = self.regs.lock().unwrap(); - if saved[0] != regs[0] { - return Err(format!("pci: saved header is for {:#010x}, this function is {:#010x}", saved[0], regs[0])); - } - *regs = saved; - Ok(()) - } -} - -/// A PCI function, as a host bridge sees it. -/// -/// BAR accesses arrive with the offset and value the device sees: whatever -/// the bridge does to byte lanes it has already done. -pub trait PciBusDevice: Send + Sync { - /// The function's configuration header. - fn config(&self) -> &PciConfig; - - /// What a configuration read of dword `index` returns to the guest. The - /// header as stored, unless the chip decodes fewer address lines than - /// the bus gives it (IOC3 repeats its first 32 bytes). - fn config_dword(&self, index: usize) -> u32 { - self.config().read(index) - } - - /// A read of `width` bytes (1, 2 or 4) at `off` within BAR `bar`. - fn bar_read(&self, bar: usize, off: u32, width: usize) -> u32; - - fn bar_write(&self, bar: usize, off: u32, val: u32, width: usize); - - /// The bridge wires the function's interrupt pin to `line`. - fn connect_interrupt(&self, _line: IrqLine) {} - - /// The bridge gives a function that masters the bus its way to memory. - fn connect_dma(&self, _dma: std::sync::Arc) {} -} - -/// How a bus master's access ended. -#[must_use] -#[derive(Clone, Copy, Debug, PartialEq, Eq)] -pub enum PciDmaStatus { - Ok, - /// Some of the bytes had no target: reads return zero for them and - /// writes drop them. A bridge may also log it and raise its error - /// interrupt. - MasterAbort, -} - -impl PciDmaStatus { - pub fn is_ok(self) -> bool { - self == PciDmaStatus::Ok - } -} - -/// Memory as a bus master reaches it. The bridge decides what a PCI address -/// is and in which order its bytes arrive; the function sees bytes at bus -/// addresses, as it would on the bus. -/// -/// A function calls only the object `connect_dma` gave it and never learns -/// whether it is shared. The O2's MACE hands every function one object (its -/// translation is fixed); a bridge whose translation depends on the -/// mastering device, as the Octane's does, hands each its own. -pub trait PciDma: Send + Sync { - /// `buf.len()` bytes from bus address `addr`, 64 bits wide for masters - /// that use dual address cycles. - fn dma_read(&self, addr: u64, buf: &mut [u8]) -> PciDmaStatus; - - /// `data` to bus address `addr`. - fn dma_write(&self, addr: u64, data: &[u8]) -> PciDmaStatus; -} - -// ---- BARs on the physical bus ---- - -/// What a host bridge does to a CPU access on its way to a function: which -/// byte lanes carry it, and how an access wider than the bus is split. -/// The bridge implements it once per way it is configured to move bytes. -pub trait PciLanes: Send + Sync { - /// A CPU read of `size` bytes (1, 2, 4 or 8) whose first byte is at - /// offset `off` of BAR `bar` of `f`, before the bridge moves lanes. - /// The value is in CPU order, most significant byte first in memory. - fn read(&self, f: &dyn PciBusDevice, bar: usize, off: u32, size: usize) -> u64; - - fn write(&self, f: &dyn PciBusDevice, bar: usize, off: u32, size: usize, val: u64); -} - -/// A run of a host bridge's CPU window that one BAR decodes whole, as a -/// decode-table entry: the access goes straight to the function through the -/// bridge's lanes, with no search of the bus and no bridge lock. The bridge -/// builds these with [`bar_runs`] whenever what its windows reach changes, -/// and the machine maps them on its physical bus over the bridge's own -/// window; every other page of the window stays with the bridge. -pub struct PciBar { - lanes: Arc, - dev: Arc, - bar: usize, - /// The CPU address bits the bridge decodes, and where in them the BAR's - /// byte 0 sits (it may lie below the window: a BAR larger than it). - mask: u64, - base: u64, -} - -impl PciBar { - pub fn new(lanes: Arc, dev: Arc, bar: usize, mask: u64, base: u64) -> Self { - Self { lanes, dev, bar, mask, base } - } - - #[inline] - fn off(&self, addr: u64) -> u32 { - (addr & self.mask).wrapping_sub(self.base) as u32 - } - - fn read(&self, addr: u64, size: usize) -> u64 { - self.lanes.read(&*self.dev, self.bar, self.off(addr), size) - } - - fn write(&self, addr: u64, size: usize, val: u64) -> u32 { - self.lanes.write(&*self.dev, self.bar, self.off(addr), size, val); - BUS_OK - } -} - -impl BusDevice for PciBar { - fn read8(&self, a: u64) -> BusRead8 { BusRead8::ok(self.read(a, 1) as u8) } - fn read16(&self, a: u64) -> BusRead16 { BusRead16::ok(self.read(a, 2) as u16) } - fn read32(&self, a: u64) -> BusRead32 { BusRead32::ok(self.read(a, 4) as u32) } - fn read64(&self, a: u64) -> BusRead64 { BusRead64::ok(self.read(a, 8)) } - fn write8(&self, a: u64, v: u8) -> u32 { self.write(a, 1, v as u64) } - fn write16(&self, a: u64, v: u16) -> u32 { self.write(a, 2, v as u64) } - fn write32(&self, a: u64, v: u32) -> u32 { self.write(a, 4, v as u64) } - fn write64(&self, a: u64, v: u64) -> u32 { self.write(a, 8, v) } -} - -/// A run of a bridge's CPU window, `start..end`, and what answers it. -#[derive(Clone)] -pub struct BarRun { - pub start: u64, - pub end: u64, - pub dev: Arc, -} - -/// One BAR some function decodes, as [`bar_runs`] reports it to `entry`: -/// the function's slot, the function, the BAR, and the CPU address (in the -/// window's terms) of its byte 0. -pub struct BarAt<'a> { - pub slot: usize, - pub dev: &'a Arc, - pub bar: usize, - pub cpu_base: u64, -} - -/// The runs of a CPU window `cpu..cpu + len` opening onto PCI `space` at -/// PCI address `pci` that one BAR decodes whole, in whole decode-table -/// pages. `entry` makes (or finds) the table entry for each BAR. Pages a -/// BAR covers only in part, and pages two BARs claim (a misconfiguration -/// the bridge resolves its own way), are left out: the bridge decodes those -/// itself. -pub fn bar_runs( - funcs: &[(usize, Arc)], - space: Space, - cpu: u64, - pci: u32, - len: u64, - mut entry: impl FnMut(BarAt) -> Arc, -) -> Vec { - const PAGE: u64 = 1 << crate::ppmem::devmap::SLOT_SHIFT; - let (wlo, whi) = (pci as u64, pci as u64 + len); - let mut found: Vec<(u64, u64, usize, usize, u64)> = Vec::new(); - for (k, (_, f)) in funcs.iter().enumerate() { - for (sp, bar, base, size) in f.config().windows() { - if sp != space { - continue; - } - let (lo, hi) = ((base as u64).max(wlo), (base as u64 + size as u64).min(whi)); - let (lo, hi) = (lo.next_multiple_of(PAGE), hi & !(PAGE - 1)); - if lo < hi { - found.push((lo, hi, k, bar, base as u64)); - } - } - } - let overlaps = |i: usize| { - let (lo, hi, ..) = found[i]; - found.iter().enumerate().any(|(j, &(l, h, ..))| j != i && l < hi && lo < h) - }; - (0..found.len()) - .filter(|&i| !overlaps(i)) - .map(|i| { - let (lo, hi, k, bar, base) = found[i]; - let (slot, dev) = (&funcs[k].0, &funcs[k].1); - let at = |p: u64| cpu.wrapping_add(p).wrapping_sub(wlo); - BarRun { start: at(lo), end: at(hi), dev: entry(BarAt { slot: *slot, dev, bar, cpu_base: at(base) }) } - }) - .collect() -} - -/// Where a bridge's runs go: the machine's bus maps each over the bridge's -/// window and gives the pages the last call named back to the bridge. -pub type BarRunSink = Box) + Send + Sync>; - +//! The host bridge owns everything machine-specific: how configuration +//! cycles are generated, where the CPU sees PCI memory, byte order, and +//! which interrupt input each slot's pins are wired to. A function only +//! sees configuration registers, accesses to its BARs, and an interrupt +//! line for its pin. + +mod bus; +mod config; +mod decode; +mod function; +mod host; +mod intx; +mod pci_to_pci_bridge; #[cfg(test)] -mod tests { - use super::*; - - fn scsi_like() -> PciConfig { - PciConfig::new(0x9004, 0x8078, 0x0100_00, 1, - [Bar::Io(0x100), Bar::Mem(0x1000), Bar::Unused, Bar::Unused, Bar::Unused, Bar::Unused], PCI_INTA) - } - - fn scsi_like_with_policy(policy: PciConfigPolicy<'_>) -> PciConfig { - PciConfig::new_with_policy( - 0x9004, 0x8078, 0x0100_00, 1, - [Bar::Io(0x100), Bar::Mem(0x1000), Bar::Unused, Bar::Unused, Bar::Unused, Bar::Unused], PCI_INTA, - policy, - ) - } - - /// Firmware learns how much space a device wants by writing all ones to - /// a BAR and reading back which address bits stuck. - #[test] - fn save_load_round_trip() { - use crate::traits::Saveable; - let src = scsi_like(); - src.write(1, PCI_COMMAND_MEMORY | PCI_COMMAND_IO); - src.write(4, 0x0000_1000); - src.write(5, 0x8000_1000); - src.write(15, 0x0000_0108); - let v1 = src.save_state(); - let dst = scsi_like(); - dst.load_state(&v1).expect("load_state"); - assert_eq!(v1, dst.save_state(), "PciConfig save_state mismatch after load_state round-trip"); - assert_eq!(dst.bar_base(1), 0x8000_1000); - let other = PciConfig::new(0x1234, 0x5678, 0, 0, [Bar::Unused; 6], 0); - assert!(other.load_state(&v1).is_err(), "a header loads only into the same kind of function"); - } - - #[test] - fn a_bar_reports_its_size_to_a_probe() { - let c = scsi_like(); - assert_eq!(c.read(0), 0x8078_9004, "device and vendor id"); - c.write(4, 0xffff_ffff); - assert_eq!(c.read(4), 0xffff_ff01, "256 bytes of I/O space"); - c.write(5, 0xffff_ffff); - assert_eq!(c.read(5), 0xffff_f000, "4 KiB of memory space"); - // An unimplemented BAR reads back as zero however hard it is probed. - c.write(6, 0xffff_ffff); - assert_eq!(c.read(6), 0); - c.write(5, 0x8000_1000); - assert_eq!(c.bar_base(1), 0x8000_1000, "and then holds the address assigned to it"); - } - - /// A function decodes a space only once it has been told it may, and - /// only within the BAR it was assigned. - #[test] - fn a_function_claims_only_its_own_window_and_only_when_enabled() { - let c = scsi_like(); - c.write(5, 0x8000_1000); - assert_eq!(c.decode(Space::Mem, 0x8000_1000), None, "memory decoding is off until enabled"); - c.write(1, PCI_COMMAND_MEMORY); - assert_eq!(c.decode(Space::Mem, 0x8000_1084), Some((1, 0x84))); - assert_eq!(c.decode(Space::Mem, 0x8000_2000), None, "past the end of the BAR"); - assert_eq!(c.decode(Space::Io, 0x8000_1084), None, "and not in I/O space"); - } - - #[test] - fn reset_returns_the_header_to_power_up() { - let c = scsi_like(); - let fresh: Vec = (0..64).map(|i| c.read(i)).collect(); - c.write(1, PCI_COMMAND_MEMORY | PCI_COMMAND_IO); - c.write(4, 0x0000_1000); - c.write(5, 0x8000_1000); - c.write(15, 0x0000_0009); - c.reset(); - assert_eq!((0..64).map(|i| c.read(i)).collect::>(), fresh); - assert_eq!(c.decode(Space::Mem, 0x8000_1000), None); - assert_eq!(c.interrupt_pin(), PCI_INTA, "the pin is wiring, not state"); - } - - /// A subsystem identity set after construction, as IOC3 sets its own, - /// is the board's and survives a reset. - #[test] - fn reset_keeps_the_subsystem_identity() { - let c = scsi_like(); - c.set_subsystem_identity(0xc304_10a9); - c.write(4, 0x0000_1000); - c.reset(); - assert_eq!(c.read(11), 0xc304_10a9); - assert_eq!(c.read(4), 1); - } - - /// Firmware programs the latency timer and cache line size, and reads - /// them back; the header type byte above them stays what it is. - #[test] - fn the_latency_timer_and_cache_line_size_are_stored() { - let c = scsi_like(); - c.write(3, 0x00ff_2810); - assert_eq!(c.read(3), 0x0000_2810); - c.reset(); - assert_eq!(c.read(3), 0); - } - - #[test] - fn the_interrupt_pin_is_read_only() { - let c = scsi_like(); - c.write(15, 0x0000_0009); - assert_eq!(c.read(15) & 0xffff, 0x0109, "line 9 assigned, pin A kept"); - c.write(15, 0); - assert_eq!(c.interrupt_pin(), PCI_INTA); - } - - #[test] - fn header_policies_mask_writes_and_restore_reset_values() { - for (status, command, latency, cache) in [(0x0200, 0xffff, 0, 0), (0x02b0, 0x0156, 0xf0, 0x0f)] { - let c = scsi_like_with_policy(PciConfigPolicy { - status, - status_w1c_mask: 0, - command_mask: command, - latency_mask: latency, - cache_line_mask: cache, - registers: &[PciRegisterPolicy { - reg_dword: 18, - initial: 0xaabb_0020, - writable_mask: 0xff00, - }], - }); - c.write(1, u32::MAX); - c.write(3, u32::MAX); - c.write(18, u32::MAX); - assert_eq!(c.read(1), (u32::from(status) << 16) | u32::from(command)); - assert_eq!(c.read(3), (u32::from(latency) << 8) | u32::from(cache)); - assert_eq!(c.read(18), 0xaabb_ff20); - c.reset(); - assert_eq!(c.read(1), u32::from(status) << 16); - assert_eq!(c.read(3), 0); - assert_eq!(c.read(18), 0xaabb_0020); - } - } - - #[test] - fn masked_writes_preserve_unselected_fields_and_status_events() { - let c = scsi_like_with_policy(PciConfigPolicy { - status: 0x0200, - status_w1c_mask: 0x8040, - command_mask: 0x0157, - latency_mask: 0xff, - cache_line_mask: 0xff, - registers: &[PciRegisterPolicy { - reg_dword: 18, - initial: 0xaabb_0000, - writable_mask: 0xffff, - }], - }); - c.set_status_bits(0x8040); - c.write(1, 0x0157); - c.write(3, 0x1234); - c.write(5, 0x1234_5000); - c.write(4, 0x1201); - for (reg, value, mask, expected) in [ - (1, 0xffff_0002, 0xff, 0x8240_0102), - (1, 0xffff_0000, 0x8000_0000, 0x0240_0102), - (1, 0x0040_0000, 0x00ff_0000, 0x0200_0102), - (1, u32::MAX, 0, 0x0200_0102), - (3, 0xffff_00ab, 0xff, 0x12ab), - (5, 0xabcd_ffff, 0xffff_0000, 0xabcd_5000), - (4, 0x3400, 0xff00, 0x3401), - (15, u32::MAX, 0xff00, 0x0100), - (18, u32::MAX, 0xff, 0xaabb_00ff), - ] { - c.write_masked(reg, value, mask); - assert_eq!(c.read(reg), expected, "register {reg}"); - } - c.set_status_bits(u16::MAX); - c.reset(); - assert_eq!(c.read(1), 0x0200_0000); - let default = scsi_like(); - default.set_status_bits(u16::MAX); - assert_eq!(default.read(1), 0x0200_0000); - } - - #[test] - fn snapshots_restore_registers_with_destination_write_policies() { - use crate::traits::Saveable; - let src = scsi_like_with_policy(PciConfigPolicy { - status: 0x02b0, - status_w1c_mask: 0xc000, - command_mask: 0x0157, - latency_mask: 0xff, - cache_line_mask: 0xff, - registers: &[PciRegisterPolicy { - reg_dword: 18, - initial: 0, - writable_mask: 0x100, - }], - }); - src.set_status_bits(0x8000); - src.write(3, 0x1234); - src.write(18, 0x100); - let saved = src.save_state(); - for (events, command, cache) in [(0, 7, 0x0f), (0xc000, 0x0157, 0xff)] { - let dst = scsi_like_with_policy(PciConfigPolicy { - status: 0x0200, - status_w1c_mask: events, - command_mask: command, - latency_mask: 0, - cache_line_mask: cache, - registers: &[PciRegisterPolicy { - reg_dword: 18, - initial: 0x20, - writable_mask: 0x200, - }], - }); - dst.load_state(&saved).unwrap(); - assert_eq!(dst.save_state(), saved); - assert_eq!(dst.read(1), 0x0200_0000 | ((0x8000 & u32::from(events)) << 16)); - dst.write_masked(1, u32::MAX, u32::MAX); - dst.write(3, u32::MAX); - dst.write(18, u32::MAX); - assert_eq!(dst.read(1), 0x0200_0000 | u32::from(command)); - assert_eq!(dst.read(3), (0x1234 & !u32::from(cache)) | u32::from(cache)); - assert_eq!(dst.read(18), 0x300); - dst.reset(); - assert_eq!(dst.read(18), 0x20); - } - } - - struct Nothing; - impl BusDevice for Nothing {} - - struct Func(PciConfig); - impl PciBusDevice for Func { - fn config(&self) -> &PciConfig { &self.0 } - fn bar_read(&self, _: usize, off: u32, _: usize) -> u32 { off } - fn bar_write(&self, _: usize, _: u32, _: u32, _: usize) {} - } - - /// Whole pages of a window a BAR decodes become runs at their CPU - /// addresses; an I/O BAR smaller than a page, a page shared by two - /// BARs, and a BAR outside the window do not. - #[test] - fn bar_runs_are_the_pages_one_bar_decodes_whole() { - let a: Arc = Arc::new(Func(scsi_like())); - let b: Arc = Arc::new(Func(scsi_like())); - for (f, io, mem) in [(&a, 0x100, 0x8000_1000u32), (&b, 0x200, 0x8000_3000)] { - f.config().write(4, io); - f.config().write(5, mem); - f.config().write(1, PCI_COMMAND_IO | PCI_COMMAND_MEMORY); - } - let funcs = vec![(1, a.clone()), (2, b.clone())]; - let mut seen = Vec::new(); - let runs = bar_runs(&funcs, Space::Mem, 0x1a00_0000, 0x8000_0000, 0x200_0000, |at| { - seen.push((at.slot, at.bar, at.cpu_base)); - Arc::new(Nothing) as Arc - }); - let spans: Vec<_> = runs.iter().map(|r| (r.start, r.end)).collect(); - assert_eq!(spans, vec![(0x1a00_1000, 0x1a00_2000), (0x1a00_3000, 0x1a00_4000)]); - assert_eq!(seen, vec![(1, 1, 0x1a00_1000), (2, 1, 0x1a00_3000)]); - assert!(bar_runs(&funcs, Space::Io, 0x1800_0000, 0, 0x200_0000, |_| unreachable!()).is_empty(), "I/O BARs are smaller than a page"); - b.config().write(5, 0x8000_1000); - assert!(bar_runs(&funcs, Space::Mem, 0x1a00_0000, 0x8000_0000, 0x200_0000, |_| unreachable!()).is_empty(), "two BARs on one page"); - assert!(bar_runs(&funcs, Space::Mem, 0x1a00_0000, 0x9000_0000, 0x200_0000, |_| unreachable!()).is_empty(), "outside the window"); - } - -} +pub(crate) mod testing; + +pub use bus::PciBus; +pub use config::{Bar, PciConfig, PciConfigPolicy, PciRegisterPolicy, Space, PCI_COMMAND_IO, PCI_COMMAND_MEMORY, PCI_INTA}; +pub use decode::{BarMapping, BarMappingSink, CpuWindow}; +pub use function::{DevFn, PciDma, PciDmaStatus, PciFunction, PciFunctionState}; +pub use host::{BarDevice, BarDevices, ByteOrder, PciHost}; +pub use intx::{BoardIrq, IntxRouting}; +pub use pci_to_pci_bridge::PciToPciBridge; diff --git a/src/pci/bus.rs b/src/pci/bus.rs new file mode 100644 index 000000000..55f116203 --- /dev/null +++ b/src/pci/bus.rs @@ -0,0 +1,805 @@ +//! One PCI bus: the functions on it, the BAR regions they decode, and the +//! interrupt pins they raise. +//! +//! A host bridge (MACE, the Octane's Bridge) owns a [`PciBus`] and supplies, as +//! a [`PciHost`], only what is its own: the DMA a requester gets and where each +//! host IRQ goes. It pushes its CPU windows with [`PciBus::set_host_windows`] +//! and decides what byte order to apply to an access. The bus has no byte +//! order, no SCSI and no card names; a machine fits its functions with +//! [`PciBus::attach`] and the bus gives each its interrupt line and its way to +//! memory. +//! +//! Headers change only through the bus (`config_write`, `reset`, a snapshot +//! load), and each change updates the mappings, so a stale BAR mapping +//! cannot outlive the header that made it. A function never resets its own +//! header. + +use std::io::Write; +use std::sync::atomic::{AtomicU8, Ordering}; +use std::sync::{Arc, Weak}; + +use super::decode::{BarMappingSink, BarRegion, CpuWindow, Decode}; +use super::intx::{BoardIrq, Combiner, IntxRouting}; +use super::{DevFn, PciDma, PciFunction, PciHost, Space}; +use crate::devlog::LogModule; +use crate::irq::IrqLine; +use crate::snapshot::{get_field, hex_u32, hex_u64, toml_u32, toml_u64}; +use crate::traits::Saveable; + +/// Header type bit 7 (dword 3 bit 23): more functions follow at this device. +const HEADER_MULTIFUNCTION: u32 = 1 << 23; + +/// A bus handle. Cheap to clone; the host holds the long-lived one. +#[derive(Clone)] +pub struct PciBus(Arc); + +/// What is in front of a bus: the host, for a root bus, or the bridge +/// function it sits behind. +enum Uplink { + Root { + up: Weak, + wiring: &'static IntxRouting, + combiner: Arc, + /// The next iris bus number to hand a secondary bus. + next: AtomicU8, + }, + /// Held weakly: the parent holds the bridge function, which holds this + /// bus. + Behind { parent: Weak, bridge: DevFn }, +} + +struct Inner { + number: u8, + uplink: Uplink, + decode: Decode, +} + + +impl PciBus { + /// A bus numbered `number` (iris's own, from 0 per machine) wired as + /// `wiring`, hosted by `up`. Build it inside `Arc::new_cyclic` so `up` + /// can point at the host. + pub fn new(number: u8, wiring: &'static IntxRouting, up: Weak) -> Self { + Self(Arc::new(Inner { + number, + decode: Decode::new(wiring.devices), + uplink: Uplink::Root { combiner: Combiner::new(wiring, up.clone()), up, wiring, next: AtomicU8::new(number + 1) }, + })) + } + + /// The bus behind the PCI-to-PCI bridge function that will sit at `bridge` + /// on `parent`. It has no host and no outputs: its functions' DMA is the + /// bridge's (so a function behind a bridge inherits the root slot the + /// bridge sits in), and a pin of device `d` reaches the parent's pin + /// `(pin + d) mod 4` of the bridge's device. Its regions join the + /// parent's table while the bridge's header forwards them. + pub fn secondary(parent: PciBus, bridge: DevFn) -> Self { + Self(Arc::new(Inner { + number: parent.next_number(), + decode: Decode::new(32), + uplink: Uplink::Behind { parent: Arc::downgrade(&parent.0), bridge }, + })) + } + + /// This bus's regions and those behind its bridges, for the bus in front. + pub(super) fn all_regions(&self) -> Vec { + self.0.decode.all_regions() + } + + pub fn number(&self) -> u8 { + self.0.number + } + + /// The next iris bus number: the root bus counts them for its tree. + fn next_number(&self) -> u8 { + match &self.0.uplink { + Uplink::Root { next, .. } => next.fetch_add(1, Ordering::Relaxed), + Uplink::Behind { parent, .. } => parent.upgrade().map_or(0, |p| PciBus(p).next_number()), + } + } + + /// The DMA of a function on this bus: the host's for the root slot the + /// function sits in, or behind, so a bridge's children are masters as + /// the bridge's slot. + fn dma_of(&self, requester: u8) -> Arc { + match &self.0.uplink { + Uplink::Root { up, .. } => up.upgrade().expect("pci: the host is gone").dma_for(requester), + Uplink::Behind { parent, bridge } => PciBus(parent.upgrade().expect("pci: the parent bus is gone")).dma_of(bridge.device), + } + } + + /// The line device `dev`'s pin (1 = INTA) drives, if it is routed: at + /// the root through the wiring, behind a bridge by the bridge's own + /// swizzle onto the pins of the bridge's device. + fn pin_line(&self, dev: u8, pin: u8) -> Option { + match &self.0.uplink { + Uplink::Root { combiner, .. } => combiner.pin_line(dev, pin), + Uplink::Behind { parent, bridge } => { + let parent = PciBus(parent.upgrade()?); + parent.pin_line(bridge.device, (pin - 1 + dev) % 4 + 1) + } + } + } + + /// Update the decode table and the BAR mappings here and on every bus + /// in front of this one, which merges this bus's regions. + fn update_mappings(&self, make: Option<&dyn Fn() -> Vec>) { + self.0.decode.update_mappings(make); + if let Uplink::Behind { parent, .. } = &self.0.uplink { + if let Some(parent) = parent.upgrade() { + PciBus(parent).update_mappings(None); + } + } + } + + /// Fit a function: give it its way to memory and, if it has an interrupt + /// pin, the line that pin drives, then tell it it is attached. Panics, as a + /// wiring mistake in the machine, on a device number the wiring lacks, an + /// address already taken, or a pin the wiring does not route. + pub fn attach(&self, at: DevFn, f: Arc) { + let i = &*self.0; + assert!(at.device < i.decode.devices(), "pci: bus {} has no device {}", i.number, at.device); + let pin = f.state().config().interrupt_pin(); + let line = (pin != 0).then(|| { + self.pin_line(at.device, pin).unwrap_or_else(|| { + panic!("pci: bus {} does not route INT{} of device {}", i.number, (b'A' + pin - 1) as char, at.device) + }) + }); + f.state().connect_dma(self.dma_of(at.device)); + if let Some(line) = line { + f.state().connect_irq(line); + } + f.attached(); + assert!(i.decode.insert(at, f), "pci: bus {} device {} function {} is taken", i.number, at.device, at.function); + self.update_mappings(None); + } + + pub fn function(&self, at: DevFn) -> Option<&Arc> { + self.0.decode.function(at) + } + + /// Whether any function sits at device number `device`. + pub fn occupied(&self, device: u8) -> bool { + (0..8).any(|function| self.function(DevFn { device, function }).is_some()) + } + + /// A configuration read of `width` bytes (1, 2 or 4) at byte `off`. + /// None: nothing answers on a root bus, and the host decides what that + /// looks like. On a bus behind a PCI-to-PCI bridge the cycle ends in + /// master abort at the bridge: it reads all ones and the bridge's + /// secondary status records the abort. + /// + /// Function 0 of a device with a function above it reports the + /// multifunction bit of the header type, so a scan visits the rest. + pub fn config_read(&self, at: DevFn, off: usize, width: usize) -> Option { + let mask = u32::MAX >> (32 - 8 * width); + let val = match self.function(at) { + Some(f) => Some((self.header_dword(at, f, off / 4) >> ((off & 3) * 8)) & mask), + None => self.master_abort().then_some(mask), + }; + crate::dlog!(LogModule::Pci, "pci: bus {} {}.{} read {off:#04x}/{width} -> {}", self.0.number, at.device, at.function, + val.map_or("no answer".to_string(), |v| format!("{v:#x}"))); + val + } + + /// Dword `index` of `f`'s header as a guest reads it: header type bit 7 + /// is the bus's to say, since only the bus knows whether another + /// function shares the device. + fn header_dword(&self, at: DevFn, f: &Arc, index: usize) -> u32 { + let multifunction = index == 3 && at.function == 0 && (1..8).any(|function| self.function(DevFn { function, ..at }).is_some()); + f.config_dword(index) | if multifunction { HEADER_MULTIFUNCTION } else { 0 } + } + + /// A cycle that no function on this bus claims. Behind a bridge it + /// ends in master abort: the bridge returns all ones for a read and + /// drops a write, and sets received master abort in its secondary + /// status. True when this bus handled it so. A root bus leaves the + /// outcome to its host. + fn master_abort(&self) -> bool { + let Uplink::Behind { parent, bridge } = &self.0.uplink else { return false }; + if let Some(parent) = parent.upgrade() { + if let Some(f) = PciBus(parent).function(*bridge) { + f.state().config().record_master_abort(); + } + } + true + } + + /// A configuration write of `width` bytes at byte `off`: only those bytes + /// change, and the header decides which of their bits stick. False if + /// nothing answers on a root bus; behind a bridge the write is discarded by + /// master abort, as a read of nothing is (see [`PciBus::config_read`]), and + /// counts as answered. The mappings are updated when the command register, + /// a BAR, or a bridge's bus numbers or windows changed, the only registers + /// they are built from. + pub fn config_write(&self, at: DevFn, off: usize, width: usize, val: u32) -> bool { + let f = self.function(at); + crate::dlog!(LogModule::Pci, "pci: bus {} {}.{} write {off:#04x}/{width} {val:#x}{}", self.0.number, at.device, at.function, + if f.is_some() { "" } else { ", no answer" }); + let Some(f) = f else { return self.master_abort() }; + let (cfg, reg, shift) = (f.state().config(), off / 4, (off & 3) * 8); + let mask = (u32::MAX >> (32 - 8 * width)) << shift; + let old = cfg.read(reg); + cfg.write_masked(reg, val << shift, mask); + let new = cfg.read(reg); + if new != old { + f.config_written(reg, old, new); + // The command register, the BARs and, in a bridge's header, its + // bus numbers and windows: the registers the table is built from. + if matches!(reg, 1 | 4..=9) { + self.update_mappings(None); + } + } + true + } + + /// The function, BAR and BAR offset that PCI address `addr` reaches in + /// `space`: the host's slow path. A borrow, no clone. + pub fn decode(&self, space: Space, addr: u32) -> Option<(&dyn PciFunction, usize, u32)> { + self.0.decode.decode(space, addr) + } + + /// The host's CPU windows changed (MACE's window registers, the Bridge's + /// Device(x)): `make` returns them, and the runs are published again. + /// `make` runs inside the publish mutex, so two CPUs writing registers + /// cannot publish out of order. The host must not hold a lock `make` + /// takes when it calls this; the order is publish mutex, then the + /// host's lock. + pub fn set_host_windows(&self, make: &dyn Fn() -> Vec) { + self.0.decode.update_mappings(Some(make)); + } + + /// Where the runs of the CPU windows that one BAR decodes whole go. The + /// machine maps them on the physical bus over the host's windows. + pub fn on_bar_mappings(&self, sink: BarMappingSink) { + self.0.decode.set_mapping_sink(sink); + } + + /// The line board IRQ `w` drives into the host. + pub fn board_irq(&self, w: BoardIrq) -> IrqLine { + match &self.0.uplink { + Uplink::Root { combiner, .. } => combiner.board_irq_line(w), + Uplink::Behind { .. } => panic!("pci: a bus behind a bridge has no board IRQs"), + } + } + + /// RST#: each function in (device, function) order has its header + /// reset and then [`PciFunction::reset`] called, and the mappings are + /// updated. Pin levels are left alone: a function that was not reset + /// keeps its line, as its hardware would, and a function that was drops + /// its own. + pub fn reset(&self) { + self.reset_where(|_| true); + } + + /// RST# to the functions at device number `device` alone, as `reset` + /// does for the whole bus. For a host with a reset line per device. + pub fn reset_device(&self, device: u8) { + self.reset_where(|at| at.device == device); + } + + fn reset_where(&self, hit: impl Fn(DevFn) -> bool) { + for (_, f) in self.0.decode.functions().filter(|&(at, _)| hit(at)) { + f.state().config().reset(); + f.reset(); + } + self.update_mappings(None); + } + + /// Tell the host every output's level again, whatever it already has. + /// A restore ends with this, after the host and the interrupt + /// controller behind it have loaded. + pub fn redrive(&self) { + if let Uplink::Root { combiner, .. } = &self.0.uplink { + combiner.redrive(); + } + } + + /// Each output's (dirty, fell, last), for a test to compare. + #[cfg(test)] + pub(crate) fn delivery_flags(&self) -> Vec<(bool, bool, bool)> { + match &self.0.uplink { + Uplink::Root { combiner, .. } => combiner.flags(), + Uplink::Behind { .. } => Vec::new(), + } + } + + /// Latch every output's `fell` and `dirty`, as an interrupted delivery + /// would leave them. + #[cfg(test)] + pub(crate) fn latch_delivery_flags(&self) { + if let Uplink::Root { combiner, .. } = &self.0.uplink { + combiner.out_force_flags(); + } + } + + /// One line per function: address, identity, command, assigned BARs and + /// where its interrupt pin goes. + pub fn describe(&self, w: &mut dyn Write) { + for (at, f) in self.0.decode.functions() { + let c = f.state().config(); + let id = c.read(0); + let bars: Vec = (0..6).map(|i| c.bar_base(i)).filter(|&b| b != 0).map(|b| format!("{b:#x}")).collect(); + let pin = c.interrupt_pin(); + let irq = match pin { + 0 => "no interrupt".to_string(), + p => { + let name = (b'A' + p - 1) as char; + match &self.0.uplink { + Uplink::Root { combiner, .. } => match combiner.output(at.device, p) { + Some(o) => format!("INT{name} to output {o}"), + None => format!("INT{name} unrouted"), + }, + Uplink::Behind { .. } if self.pin_line(at.device, p).is_some() => format!("INT{name} through the bridge"), + Uplink::Behind { .. } => format!("INT{name} unrouted"), + } + } + }; + let _ = writeln!(w, " {}.{}: {:04x}:{:04x}, command {:#06x}, BARs [{}], {irq}", + at.device, at.function, id & 0xffff, id >> 16, c.command(), bars.join(", ")); + if let Some(sec) = f.secondary() { + let _ = writeln!(w, " bus {} behind {}.{}:", sec.number(), at.device, at.function); + sec.describe(w); + } + } + } + + /// The bus a guest's bus number reaches: bus 0 is this (root) bus, and + /// any other number is found by the bridges' secondary and subordinate + /// bus numbers as firmware has programmed them, however deep. None if + /// no bridge's range holds it. A scan, for configuration cycles. + pub fn find_bus(&self, guest_bus: u8) -> Option { + if guest_bus == 0 { + return Some(self.clone()); + } + self.find_behind(guest_bus) + } + + /// The bus a type 1 configuration cycle for `guest_bus` on this bus + /// reaches. Type 0 functions ignore a type 1 cycle; a PCI-to-PCI bridge + /// claims it when the number is in its secondary to subordinate range. + /// None when no bridge does. Unlike [`PciBus::find_bus`], bus 0 is not this + /// bus: it too reaches only a bridge whose range holds it. + pub fn type1_target(&self, guest_bus: u8) -> Option { + self.find_behind(guest_bus) + } + + /// The bus numbered `guest_bus` among the buses behind this one's + /// bridges (number 0 is never behind a bridge). + fn find_behind(&self, guest_bus: u8) -> Option { + self.0.decode.functions().find_map(|(_, f)| { + let sec = f.secondary()?; + let numbers = f.state().config().read(6); + let (first, last) = ((numbers >> 8 & 0xff) as u8, (numbers >> 16 & 0xff) as u8); + match guest_bus { + n if n == first => Some(sec.clone()), + n if first < n && n <= last => sec.find_behind(n), + _ => None, + } + }) + } +} + +impl PciBus { + /// The `pci` monitor command both hosts share: `status` describes the + /// functions, `config ` dumps one's header. + pub fn command(&self, args: &[&str], w: &mut dyn Write) -> Result<(), String> { + match args.first().copied().unwrap_or("status") { + "status" => self.describe(w), + "config" => { + let device: u8 = args.get(1).and_then(|v| v.parse().ok()).ok_or("Usage: pci config ")?; + let f = self.function(DevFn { device, function: 0 }).ok_or(format!("pci: nothing in slot {device}"))?; + for row in 0..16 { + let v: Vec = (0..4).map(|k| format!("{:08x}", f.state().config().read(row * 4 + k))).collect(); + let _ = writeln!(w, " {:02x}: {}", row * 16, v.join(" ")); + } + } + _ => return Err("Usage: pci status | config ".to_string()), + } + Ok(()) + } +} + +/// Every function's header, and the pin and board IRQ levels. A restore +/// replaces all of it and forgets what the host was last told; the machine then +/// calls [`PciBus::redrive`]. The functions behind the headers save their own +/// state. +impl Saveable for PciBus { + fn save_state(&self) -> toml::Value { + let i = &*self.0; + let mut t = toml::map::Map::new(); + let mut funcs = toml::map::Map::new(); + for (at, f) in i.decode.functions() { + funcs.insert(format!("{}.{}", at.device, at.function), f.state().config().save_state()); + } + t.insert("functions".into(), toml::Value::Table(funcs)); + // The buses behind bridges save their headers here too. + let mut behind = toml::map::Map::new(); + for (at, f) in i.decode.functions() { + if let Some(sec) = f.secondary() { + behind.insert(format!("{}.{}", at.device, at.function), sec.save_state()); + } + } + if !behind.is_empty() { + t.insert("behind".into(), toml::Value::Table(behind)); + } + if let Uplink::Root { combiner, .. } = &i.uplink { + let (lo, hi, wires) = combiner.save(); + t.insert("pins_lo".into(), hex_u64(lo)); + t.insert("pins_hi".into(), hex_u64(hi)); + t.insert("wires".into(), hex_u32(wires)); + } + toml::Value::Table(t) + } + + fn load_state(&self, v: &toml::Value) -> Result<(), String> { + let i = &*self.0; + let saved = get_field(v, "functions").and_then(|f| f.as_table()).ok_or("pci: saved state has no functions")?; + let ours: Vec = i.decode.functions().map(|(at, _)| format!("{}.{}", at.device, at.function)).collect(); + let same = ours.len() == saved.len() && ours.iter().all(|k| saved.contains_key(k)); + if !same { + return Err(format!("pci: saved state has functions [{}], this bus [{}]", + saved.keys().cloned().collect::>().join(", "), ours.join(", "))); + } + for (at, f) in i.decode.functions() { + let key = format!("{}.{}", at.device, at.function); + f.state().config().load_state(&saved[&key])?; + if let Some(sec) = f.secondary() { + let state = get_field(v, "behind").and_then(|b| get_field(b, &key)).ok_or_else(|| format!("pci: saved state has no bus behind {key}"))?; + sec.load_state(state)?; + } + } + if let Uplink::Root { combiner, .. } = &i.uplink { + let word = |k: &str| get_field(v, k).and_then(toml_u64).unwrap_or(0); + combiner.load((word("pins_lo"), word("pins_hi"), get_field(v, "wires").and_then(toml_u32).unwrap_or(0))); + } + self.update_mappings(None); + Ok(()) + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::pci::testing::{TestCard, TestHost}; + use crate::pci::{Bar, PciConfig, PCI_COMMAND_MEMORY, PCI_INTA}; + use std::sync::Mutex; + + // Device 1: INTA and INTB on outputs 8 and 9; device 2 INTA shares 8; + // device 3 INTA..INTD on 10 to 13. One board IRQ, on output 14. + static W: IntxRouting = IntxRouting { + devices: 8, + irqs: 16, + intx: &[ + (1, [Some(8), Some(9), None, None]), + (2, [Some(8), None, None, None]), + (3, [Some(10), Some(11), Some(12), Some(13)]), + ], + board_irqs: &[14], + }; + + fn at(device: u8, function: u8) -> DevFn { + DevFn { device, function } + } + + fn bus() -> (Arc, PciBus) { + let h = Arc::new(TestHost::default()); + let up: Weak = Arc::downgrade(&(h.clone() as Arc)); + (h, PciBus::new(0, &W, up)) + } + + fn bars() -> [Bar; 6] { + [Bar::Io(0x100), Bar::Mem(0x1000), Bar::Unused, Bar::Unused, Bar::Unused, Bar::Unused] + } + + #[test] + fn attach_connects_the_right_line_and_dma() { + let (h, b) = bus(); + let c = TestCard::new(bars(), PCI_INTA); + b.attach(at(3, 0), c.clone()); + assert_eq!(c.state().irq().unwrap().line(), 3 * 4); + c.state().irq().unwrap().set(true); + assert_eq!(h.irqs(), vec![(10, true)]); + let mut buf = [0u8; 2]; + assert!(c.state().dma().unwrap().dma_read(0, &mut buf).is_ok()); + assert_eq!(buf, [3, 3], "the DMA of requester device 3"); + let none = TestCard::new(bars(), 0); + b.attach(at(4, 0), none.clone()); + assert!(none.state().irq().is_none(), "no pin, no line"); + assert!(none.state().dma().is_some()); + } + + #[test] + #[should_panic(expected = "is taken")] + fn duplicate_address_panics() { + let (_h, b) = bus(); + b.attach(at(1, 0), TestCard::new(bars(), PCI_INTA)); + b.attach(at(1, 0), TestCard::new(bars(), PCI_INTA)); + } + + #[test] + #[should_panic(expected = "has no device 8")] + fn device_out_of_range_panics() { + let (_h, b) = bus(); + b.attach(at(8, 0), TestCard::new(bars(), 0)); + } + + #[test] + #[should_panic(expected = "does not route INTC of device 1")] + fn pin_without_a_route_panics() { + bus().1.attach(at(1, 0), TestCard::new(bars(), PCI_INTA + 2)); + } + + #[test] + fn multi_function_shares_the_device_pins() { + let (h, b) = bus(); + let (f0, f1) = (TestCard::new(bars(), PCI_INTA), TestCard::new(bars(), PCI_INTA)); + b.attach(at(3, 0), f0.clone()); + b.attach(at(3, 1), f1.clone()); + assert_eq!(f0.state().irq().unwrap().line(), f1.state().irq().unwrap().line()); + f0.state().irq().unwrap().set(true); + f1.state().irq().unwrap().set(true); + f0.state().irq().unwrap().set(false); + assert_eq!(h.irqs(), vec![(10, true), (10, false)], "one pin, one output"); + assert!(b.occupied(3) && !b.occupied(4)); + } + + #[test] + fn config_size_probe_and_masks() { + let (_h, b) = bus(); + b.attach(at(1, 0), TestCard::new(bars(), PCI_INTA)); + assert_eq!(b.config_read(at(1, 0), 0, 4), Some(0x5678_1234)); + assert_eq!(b.config_read(at(1, 0), 0, 2), Some(0x1234), "vendor"); + assert_eq!(b.config_read(at(1, 0), 2, 2), Some(0x5678), "device"); + assert_eq!(b.config_read(at(1, 0), 3, 1), Some(0x56)); + assert!(b.config_write(at(1, 0), 0x10, 4, 0xffff_ffff)); + assert_eq!(b.config_read(at(1, 0), 0x10, 4), Some(0xffff_ff01), "256 bytes of I/O"); + assert!(b.config_write(at(1, 0), 0x14, 4, 0xffff_ffff)); + assert_eq!(b.config_read(at(1, 0), 0x14, 4), Some(0xffff_f000), "4 KiB of memory"); + } + + /// The read-modify-write every host did before the bus did it. + #[test] + fn sub_dword_writes_equal_the_old_read_modify_write() { + for width in [1usize, 2, 4] { + for off in (0..256).step_by(width) { + for val in [0xffff_ffffu32, 0xa5c3_1e78, 0] { + let (_h, b) = bus(); + let (new, old) = (TestCard::new(bars(), PCI_INTA), PciConfig::new(0x1234, 0x5678, 0, 1, bars(), PCI_INTA)); + b.attach(at(1, 0), new.clone()); + // Start both from the same non-trivial header. + for r in 1..16 { + new.state().config().write(r, 0x1111_1111 * r as u32); + old.write(r, 0x1111_1111 * r as u32); + } + assert!(b.config_write(at(1, 0), off, width, val)); + let (i, shift) = (off / 4, 8 * (off & 3)); + let mask = (u64::MAX >> (64 - 8 * width)) << shift; + old.write(i, ((old.read(i) as u64 & !mask) | ((val as u64) << shift & mask)) as u32); + for r in 0..64 { + assert_eq!(new.state().config().read(r), old.read(r), "width {width} off {off:#x} val {val:#x} reg {r}"); + } + } + } + } + } + + #[test] + fn an_absent_function_reads_none() { + let (_h, b) = bus(); + b.attach(at(1, 0), TestCard::new(bars(), 0)); + assert_eq!(b.config_read(at(2, 0), 0, 4), None); + assert_eq!(b.config_read(at(1, 1), 0, 4), None, "function 1 of a single-function device"); + assert!(!b.config_write(at(2, 0), 0, 4, 0)); + } + + #[test] + fn a_config_write_that_moves_a_bar_updates_the_mappings() { + let (_h, b) = bus(); + b.attach(at(1, 0), TestCard::new(bars(), 0)); + let runs = Arc::new(Mutex::new(Vec::new())); + let sink = runs.clone(); + b.on_bar_mappings(Box::new(move |r| *sink.lock().unwrap() = r)); + b.set_host_windows(&|| vec![CpuWindow { space: Space::Mem, cpu: 0x1a00_0000, pci: 0x8000_0000, len: 0x100_0000, mask: 0xffff_ffff }]); + let cpus = || runs.lock().unwrap().iter().map(|r: &crate::pci::BarMapping| r.cpu).collect::>(); + assert!(cpus().is_empty()); + b.config_write(at(1, 0), 0x14, 4, 0x8000_2000); + b.config_write(at(1, 0), 0x04, 2, PCI_COMMAND_MEMORY); + assert_eq!(cpus(), vec![0x1a00_2000]); + b.config_write(at(1, 0), 0x14, 4, 0x8000_5000); + assert_eq!(cpus(), vec![0x1a00_5000], "the old run is gone"); + assert!(b.decode(Space::Mem, 0x8000_2000).is_none()); + assert_eq!(b.decode(Space::Mem, 0x8000_5010).map(|(_, bar, off)| (bar, off)), Some((1, 0x10))); + } + + /// Only the command register and the BARs shape the decode table, so a + /// write to any other register, or one that changes nothing, publishes + /// nothing. + #[test] + fn only_a_command_or_bar_change_updates_the_mappings() { + let (_h, b) = bus(); + b.attach(at(1, 0), TestCard::new(bars(), 0)); + let count = Arc::new(std::sync::atomic::AtomicUsize::new(0)); + let n = count.clone(); + b.on_bar_mappings(Box::new(move |_| { + n.fetch_add(1, std::sync::atomic::Ordering::SeqCst); + })); + let published = || count.load(std::sync::atomic::Ordering::SeqCst); + let before = published(); + for line in 0..=255u32 { + b.config_write(at(1, 0), 0x3c, 1, line); // interrupt line + b.config_write(at(1, 0), 0x0c, 1, line); // cache line size + } + assert_eq!(published(), before); + b.config_write(at(1, 0), 0x04, 2, PCI_COMMAND_MEMORY); + assert_eq!(published(), before + 1, "the command register"); + b.config_write(at(1, 0), 0x04, 2, PCI_COMMAND_MEMORY); + assert_eq!(published(), before + 1, "the same value again"); + b.config_write(at(1, 0), 0x14, 4, 0x8000_2000); + assert_eq!(published(), before + 2, "a BAR"); + } + + /// RST# resets the header and then the function, device by device. + #[test] + fn reset_orders_header_then_function_by_device_and_function() { + let (_h, b) = bus(); + let log = Arc::new(Mutex::new(Vec::new())); + for (name, d, f) in [("c", 3, 0), ("a", 1, 1), ("b", 1, 0)] { + let c = TestCard::logged(name, bars(), 0, &log); + b.attach(at(d, f), c); + b.config_write(at(d, f), 4, 2, PCI_COMMAND_MEMORY); + } + b.reset(); + assert_eq!(*log.lock().unwrap(), vec!["b cmd=0x0", "a cmd=0x0", "c cmd=0x0"], "header already reset when reset() runs"); + } + + #[test] + fn reset_device_resets_the_functions_at_one_device_number_only() { + let (_h, b) = bus(); + let log = Arc::new(Mutex::new(Vec::new())); + for (name, d, f) in [("a0", 1, 0), ("a1", 1, 1), ("b", 2, 0), ("c", 3, 0)] { + b.attach(at(d, f), TestCard::logged(name, bars(), 0, &log)); + b.config_write(at(d, f), 4, 2, PCI_COMMAND_MEMORY); + } + b.reset_device(1); + assert_eq!(*log.lock().unwrap(), vec!["a0 cmd=0x0", "a1 cmd=0x0"]); + assert_eq!(b.config_read(at(2, 0), 4, 2), Some(PCI_COMMAND_MEMORY), "device 2 keeps its header"); + assert_eq!(b.config_read(at(1, 1), 4, 2), Some(0)); + } + + #[test] + fn reset_leaves_pin_levels_until_the_card_lowers_its_line() { + let (h, b) = bus(); + let c = TestCard::new(bars(), PCI_INTA); + b.attach(at(3, 0), c.clone()); + c.state().irq().unwrap().set(true); + b.reset(); + assert_eq!(h.irqs(), vec![(10, true)], "the bus did not lower it"); + c.state().irq().unwrap().set(false); + assert_eq!(h.irqs(), vec![(10, true), (10, false)]); + } + + #[test] + fn redrive_after_load_state_resends_a_high_output() { + let (h, b) = bus(); + let c = TestCard::new(bars(), PCI_INTA); + b.attach(at(3, 0), c.clone()); + c.state().irq().unwrap().set(true); + let saved = b.save_state(); + b.load_state(&saved).unwrap(); + assert_eq!(h.irqs().len(), 1, "load tells the host nothing"); + b.redrive(); + let heard: Vec<_> = h.irqs().into_iter().filter(|o| o.0 == 10).collect(); + assert_eq!(heard, vec![(10, true), (10, true)], "the host hears the saved level again"); + } + + #[test] + fn bus_state_round_trips() { + let make = || { + let (h, b) = bus(); + let c = TestCard::new(bars(), PCI_INTA); + b.attach(at(3, 0), c.clone()); + (h, b, c) + }; + let (_ha, a, ca) = make(); + a.config_write(at(3, 0), 0x14, 4, 0x8000_1000); + ca.state().irq().unwrap().set(true); + a.board_irq(BoardIrq(0)).set(true); + let v = a.save_state(); + let (_hb, b, cb) = make(); + b.load_state(&v).unwrap(); + assert_eq!(b.save_state(), v); + assert_eq!(cb.state().config().bar_base(1), 0x8000_1000); + } + + #[test] + fn load_state_replaces_headers_pins_and_combiner_flags_outright() { + let make = || { + let (h, b) = bus(); + let c = TestCard::new(bars(), PCI_INTA); + b.attach(at(3, 0), c.clone()); + (h, b, c) + }; + let (_hs, src, cs) = make(); + src.config_write(at(3, 0), 0x14, 4, 0x8000_1000); + cs.state().irq().unwrap().set(true); + let v = src.save_state(); + let (_hf, fresh, _) = make(); + fresh.load_state(&v).unwrap(); + // A bus that has been used: other header, other pins, a latched fall. + let (_hd, dirty, cd) = make(); + dirty.config_write(at(3, 0), 0x14, 4, 0x9000_0000); + dirty.config_write(at(3, 0), 0x04, 2, PCI_COMMAND_MEMORY); + cd.state().irq().unwrap().set(true); + cd.state().irq().unwrap().set(false); + dirty.board_irq(BoardIrq(0)).set(true); + dirty.latch_delivery_flags(); + dirty.load_state(&v).unwrap(); + assert_eq!(dirty.save_state(), fresh.save_state()); + assert_eq!(dirty.delivery_flags(), fresh.delivery_flags()); + assert_eq!(dirty.config_read(at(3, 0), 4, 2), fresh.config_read(at(3, 0), 4, 2)); + } + + #[test] + fn a_mismatched_function_set_is_refused_by_load_state() { + let (_h, a) = bus(); + a.attach(at(1, 0), TestCard::new(bars(), 0)); + a.attach(at(2, 0), TestCard::new(bars(), 0)); + let (_h, b) = bus(); + b.attach(at(1, 0), TestCard::new(bars(), 0)); + let err = b.load_state(&a.save_state()).unwrap_err(); + assert!(err.contains("1.0, 2.0") && err.contains("[1.0]"), "{err}"); + } + + #[test] + fn describe_prints_one_line_per_function() { + let (_h, b) = bus(); + b.attach(at(1, 0), TestCard::new(bars(), PCI_INTA)); + b.attach(at(3, 0), TestCard::new(bars(), 0)); + b.config_write(at(1, 0), 0x14, 4, 0x8000_1000); + let mut out = Vec::new(); + b.describe(&mut out); + let out = String::from_utf8(out).unwrap(); + assert_eq!(out.lines().count(), 2); + assert!(out.lines().next().unwrap().contains("1.0: 1234:5678") && out.contains("BARs [0x80001000]") && out.contains("INTA to output 8"), "{out}"); + assert!(out.lines().nth(1).unwrap().contains("no interrupt")); + } + + #[test] + fn function_0_reports_multifunction_only_while_a_higher_function_is_fitted() { + let (_host, bus) = bus(); + let header = |function| bus.config_read(at(2, function), 0x0e, 1).unwrap(); + let card = || TestCard::new(bars(), 0); + bus.attach(at(2, 0), card()); + assert_eq!(header(0) & 0x80, 0, "a single-function device"); + bus.attach(at(2, 1), card()); + assert_eq!(header(0) & 0x80, 0x80, "function 0 tells a scan to visit function 1"); + assert_eq!(header(1) & 0x80, 0, "the bit is function 0's alone"); + assert_eq!(bus.config_read(at(2, 0), 0x0c, 4).unwrap() >> 16 & 0xff, 0x80, "in the dword too"); + bus.attach(at(3, 0), card()); + assert_eq!(bus.config_read(at(3, 0), 0x0e, 1), Some(0), "another device is unaffected"); + } + + #[test] + fn behind_a_bridge_an_unclaimed_config_cycle_reads_ones_and_a_write_is_dropped() { + use crate::pci::PciToPciBridge; + let (_host, bus) = bus(); + let bridge = PciToPciBridge::new(&bus, at(3, 0), 0x1234, 0x5678); + bus.attach(at(3, 0), bridge.clone()); + let sec = bridge.secondary(); + let status = || bus.config_read(at(3, 0), 0x1e, 2).unwrap(); + assert_eq!(status() & 0x2000, 0); + assert_eq!(sec.config_read(at(5, 0), 0, 4), Some(0xffff_ffff)); + assert_eq!(sec.config_read(at(5, 0), 2, 2), Some(0xffff)); + assert_eq!(sec.config_read(at(5, 0), 1, 1), Some(0xff)); + assert_eq!(status() & 0x2000, 0x2000, "received master abort"); + assert!(sec.config_write(at(5, 0), 4, 2, 0x2), "dropped, not unanswered"); + // Write one to clear. + bus.config_write(at(3, 0), 0x1c, 4, 0x2000_0000); + assert_eq!(status() & 0x2000, 0); + // The root bus leaves an absent function to its host. + assert_eq!(bus.config_read(at(5, 0), 0, 4), None); + assert!(!bus.config_write(at(5, 0), 4, 2, 0x2)); + assert_eq!(status() & 0x2000, 0, "the root bus's miss is not the bridge's"); + } +} diff --git a/src/pci/config.rs b/src/pci/config.rs new file mode 100644 index 000000000..e45bf14b3 --- /dev/null +++ b/src/pci/config.rs @@ -0,0 +1,705 @@ +//! A function's configuration header and its base address registers. + +use std::sync::Mutex; + +use crate::snapshot::{get_field, load_u32_slice, u32_slice_to_toml}; + +/// Command register bits. +pub const PCI_COMMAND_IO: u32 = 0x1; +pub const PCI_COMMAND_MEMORY: u32 = 0x2; + +/// Secondary status (dword 7 bit 29): received master abort. +const SEC_STATUS_MASTER_ABORT: u32 = 1 << 29; + +/// Interrupt pins, as register 0x3d holds them. +pub const PCI_INTA: u8 = 1; + +/// What one base address register decodes, and how much. +#[derive(Clone, Copy, PartialEq, Eq, Debug)] +pub enum Bar { + Unused, + /// I/O space, this many bytes (a power of two). + Io(u32), + /// 32-bit memory space, this many bytes (a power of two). + Mem(u32), + /// 64-bit memory space, this many bytes (a power of two, under 4 GiB). + /// The BAR takes two header dwords: the next entry of the array must be + /// [`Bar::Unused`], and the header makes it the upper half. Both host + /// bridges generate 32-bit PCI addresses, so a base programmed above 4 + /// GiB (a non-zero upper half) is not reachable and decodes nothing. + Mem64(u32), +} + +/// What the header holds for each BAR dword: a BAR, or the upper half of +/// the 64-bit BAR before it. +#[derive(Clone, Copy, PartialEq, Eq, Debug)] +enum BarSlot { + Bar(Bar), + Upper, +} + +/// Which address space an access is in. +#[derive(Clone, Copy, PartialEq, Eq, Debug)] +pub enum Space { + Io, + Mem, +} + +/// Initial value and writable bits for a function-specific configuration dword. +#[derive(Clone, Copy, Debug)] +pub struct PciRegisterPolicy { + /// Dword index. Header, BAR, subsystem identity, and interrupt dwords are managed separately. + pub reg_dword: usize, + /// Value restored by [`PciConfig::reset`]. + pub initial: u32, + /// Bits that configuration writes may change. + pub writable_mask: u32, +} + +/// Function-specific write behavior for a type 0 PCI configuration header. +/// +/// Register policies are copied into the header during construction; the +/// borrowed slice is not retained. +#[derive(Clone, Copy, Debug)] +pub struct PciConfigPolicy<'a> { + /// Fixed Status bits that read as one. + pub status: u16, + /// Device-latched Status bits that writes acknowledge with one. + pub status_w1c_mask: u16, + /// Writable Command bits. + pub command_mask: u16, + /// Writable bits in the latency timer byte. + pub latency_mask: u8, + /// Writable bits in the cache-line-size byte. + pub cache_line_mask: u8, + /// Initial values and write masks for function-specific dwords. + pub registers: &'a [PciRegisterPolicy], +} + +impl<'a> Default for PciConfigPolicy<'a> { + fn default() -> Self { + Self { + status: 0x0200, + status_w1c_mask: 0, + command_mask: 0xffff, + latency_mask: 0xff, + cache_line_mask: 0xff, + registers: &[], + } + } +} + +/// One function's type 0 configuration header. +/// +/// Enough of it to be discovered and configured: identity, class, command +/// and status, the interrupt pin, and base address registers that answer a +/// size probe. Writing all ones to a BAR and reading it back is how firmware +/// learns how much space a device wants, so a BAR that stores whatever it is +/// given reports a size of four gigabytes and the bus layout falls apart. +pub struct PciConfig { + regs: Mutex<[u32; 64]>, + bars: [BarSlot; 6], + /// The registers as the function powers up, for [`PciConfig::reset`]. + power_on: [u32; 64], + /// Dwords `16..caps_end` hold a capability list, which firmware cannot + /// overwrite. 0 when the function has none. + caps_end: usize, + status: u16, + status_w1c_mask: u16, + command_mask: u16, + latency_mask: u8, + cache_line_mask: u8, + writable_masks: [u32; 64], + /// A type 1 header (PCI-to-PCI bridge): no BARs, and dwords 6 to 12 are + /// bus numbers and forwarding windows. + type1: bool, +} + +impl PciConfig { + /// `class` is the 24-bit class/subclass/interface, as it sits in the top + /// three bytes of register 0x08. `pin` is 0 for none, or [`PCI_INTA`] + /// and on; the line register is left for firmware to assign. + pub fn new(vendor: u16, device: u16, class: u32, revision: u8, bars: [Bar; 6], pin: u8) -> Self { + Self::new_with_policy( + vendor, + device, + class, + revision, + bars, + pin, + PciConfigPolicy::default(), + ) + } + + /// Construct a type 0 header with function-specific register behavior. + pub fn new_with_policy( + vendor: u16, + device: u16, + class: u32, + revision: u8, + bars: [Bar; 6], + pin: u8, + policy: PciConfigPolicy<'_>, + ) -> Self { + let mut slots = bars.map(BarSlot::Bar); + for i in 0..5 { + if let Bar::Mem64(_) = bars[i] { + assert!(bars[i + 1] == Bar::Unused, "a 64-bit BAR takes the next one"); + slots[i + 1] = BarSlot::Upper; + } + } + assert!(!matches!(bars[5], Bar::Mem64(_)), "a 64-bit BAR needs a BAR after it"); + let mut r = [0u32; 64]; + r[0] = ((device as u32) << 16) | vendor as u32; + r[2] = (class << 8) | revision as u32; + r[15] = (pin as u32) << 8; + for (i, b) in bars.iter().enumerate() { + match b { + Bar::Io(_) => r[4 + i] = 1, + // Memory type 10b: decodes anywhere in 64-bit space. + Bar::Mem64(_) => r[4 + i] = 0x4, + _ => {} + } + } + let mut writable_masks = [u32::MAX; 64]; + for register in policy.registers { + assert!( + matches!(register.reg_dword, 10 | 12..=14 | 16..=63), + "pci: register policy requires a function-specific dword in 0..64"); + r[register.reg_dword] = register.initial; + writable_masks[register.reg_dword] = register.writable_mask; + } + Self { + regs: Mutex::new(r), + bars: slots, + power_on: r, + status: policy.status, + status_w1c_mask: policy.status_w1c_mask, + command_mask: policy.command_mask, + latency_mask: policy.latency_mask, + cache_line_mask: policy.cache_line_mask, + writable_masks, + caps_end: 0, + type1: false, + } + } + + /// A PCI-to-PCI bridge's type 1 header: class 0604, header type 1, no + /// BARs and no interrupt pin. Dwords 6 (primary, secondary and + /// subordinate bus numbers), 7 (16-bit I/O base and limit, bits 15:12 of + /// each), 8 (memory base and limit, bits 31:20) and 9 (32-bit + /// prefetchable memory) store what firmware writes, to the granularity + /// the register has; dwords 10 to 12, the 64-bit and 32-bit-I/O upper + /// halves, and dword 14, the expansion ROM base, read zero. The bridge + /// control register is the top half of dword 15. Everything resets to + /// zero, and the windows forward nothing until the command register + /// enables them. + pub fn type1(vendor: u16, device: u16, revision: u8) -> Self { + let mut c = Self::new(vendor, device, 0x06_0400, revision, [Bar::Unused; 6], 0); + c.type1 = true; + let r = c.regs.get_mut().unwrap(); + r[3] = 1 << 16; + c.power_on = *r; + c + } + + /// Whether this type 1 header forwards the range `base..base + size` of + /// `space` to its secondary bus: the range lies wholly inside one + /// enabled window (I/O, memory or prefetchable memory). A range that + /// straddles a window's edge is not forwarded; what a bridge does with + /// one is not defined. + pub fn forwards(&self, space: Space, base: u32, size: u32) -> bool { + let r = self.regs.lock().unwrap(); + let (lo, hi) = (base as u64, base as u64 + size as u64 - 1); + let inside = |wlo: u64, whi: u64| wlo <= whi && wlo <= lo && hi <= whi; + match space { + Space::Io => { + r[1] & PCI_COMMAND_IO != 0 && inside(((r[7] & 0xf0) as u64) << 8, ((r[7] >> 8 & 0xf0) as u64) << 8 | 0xfff) + } + Space::Mem => { + r[1] & PCI_COMMAND_MEMORY != 0 && { + let window = |dword: u32| (((dword & 0xfff0) as u64) << 16, ((dword >> 16 & 0xfff0) as u64) << 16 | 0xf_ffff); + let (mem, pf) = (window(r[8]), window(r[9])); + inside(mem.0, mem.1) || inside(pf.0, pf.1) + } + } + } + } + + /// Give the function a capability list: `caps` are the raw dwords of the + /// list, which starts at dword `first` (at least 16, the end of the + /// header). They are read-only and survive a reset. The header's + /// capability pointer and status bit are set to match. + pub fn with_capabilities(mut self, first: usize, caps: &[u32]) -> Self { + assert!(first >= 16 && first + caps.len() <= 64, "capabilities must fit in the 256-byte header"); + let r = self.regs.get_mut().unwrap(); + for (i, &d) in caps.iter().enumerate() { + r[first + i] = d; + } + // Capabilities pointer (byte offset) in the low byte of dword 13. + r[13] = (first * 4) as u32; + self.power_on = *r; + self.caps_end = first + caps.len(); + self + } + + /// RST#: back to the power-up header. The command register clears, so + /// the function stops decoding, and the BARs and line forget what + /// firmware assigned. + pub fn reset(&self) { + let mut r = self.regs.lock().unwrap(); + // The subsystem identity is the board's, set once after + // construction (IOC3's selects what a guest instantiates). + let subsystem = r[11]; + *r = self.power_on; + r[11] = subsystem; + } + + /// A type 1 header's secondary status gains received master abort: a + /// cycle forwarded to the bus behind the bridge went unclaimed. + pub fn record_master_abort(&self) { + let mut r = self.regs.lock().unwrap(); + if self.type1 { + r[7] |= SEC_STATUS_MASTER_ABORT; + } + } + + /// Register `reg` (a dword index, 0..64). + pub fn read(&self, reg: usize) -> u32 { + let r = self.regs.lock().unwrap(); + match reg { + // A function with a capability list also sets status bit 4. + 1 => (r[1] & (0xffff | ((self.status_w1c_mask as u32) << 16))) + | ((self.status as u32) << 16) + | if r[13] != 0 { 0x0010_0000 } else { 0 }, + _ => r[reg], + } + } + + pub fn write(&self, reg: usize, val: u32) { + self.write_masked(reg, val, u32::MAX); + } + + /// Write selected bits of dword `reg`. Both `val` and `mask` use the + /// register's bit positions. Unselected Status events are not acknowledged. + pub fn write_masked(&self, reg: usize, val: u32, mask: u32) { + if mask == 0 { + return; + } + let mut r = self.regs.lock().unwrap(); + match reg { + // Identity, class and header type are read-only. + 0 | 2 => {} + 3 => { + let mask = mask & (((self.latency_mask as u32) << 8) | self.cache_line_mask as u32); + r[3] = (r[3] & !mask) | (val & mask); + } + // Command writes replace selected bits; Status events clear on ones. + 1 => { + let clear = val & mask & ((self.status_w1c_mask as u32) << 16); + r[1] = (r[1] & !(mask & 0xffff) & !clear) + | (val & mask & self.command_mask as u32); + } + // A bridge's bus numbers and forwarding windows. The low nibbles + // of the base and limit registers are read-only zero (16-bit I/O + // and 32-bit memory), which also makes the upper halves of the + // prefetchable and I/O windows (dwords 10 to 12) read-only zero; + // the I/O secondary status is not kept. + 6 if self.type1 => r[reg] = (r[reg] & !mask) | (val & mask), + 10..=12 if self.type1 => {} + // I/O base and limit; the secondary status above them is + // write-one-to-clear, and only received master abort is kept. + 7 if self.type1 => { + r[reg] = (((r[reg] & !mask) | (val & mask)) & 0x0000_f0f0) | (r[reg] & !(val & mask) & SEC_STATUS_MASTER_ABORT) + } + 8 | 9 if self.type1 => r[reg] = ((r[reg] & !mask) | (val & mask)) & 0xfff0_fff0, + // A type 1 header's expansion ROM base is dword 14, and the + // bridge has no ROM either. + 14 if self.type1 => {} + 4..=9 => { + let val = (r[reg] & !mask) | (val & mask); + r[reg] = match self.bars[reg - 4] { + BarSlot::Bar(Bar::Unused) => 0, + // The upper half of a 64-bit base takes every bit. + BarSlot::Upper => val, + // Address bits above the size take the write; those + // below read back as zero, which tells firmware the + // size. The type bits stay what the BAR is. + BarSlot::Bar(Bar::Io(n)) => (val & !(n - 1)) | 1, + BarSlot::Bar(Bar::Mem(n)) => val & !(n - 1), + BarSlot::Bar(Bar::Mem64(n)) => (val & !(n - 1)) | 0x4, + }; + } + // Expansion ROM base: no function has a ROM, so the register is + // unimplemented and reads zero (PCI 2.2, 6.1 and 6.2.5.2). A size + // probe that read back its ones would have firmware map a ROM + // that is not there. + 12 => {} + // The capability list is read-only. + 16.. if reg < self.caps_end => {} + // Interrupt pin is read-only; the line is firmware's scratch. + 15 => { + let mask = mask & !0xff00; + r[15] = (r[15] & !mask) | (val & mask); + } + _ => { + let mask = mask & self.writable_masks[reg]; + r[reg] = (r[reg] & !mask) | (val & mask); + } + } + } + + /// Latch selected device Status events without replacing earlier events. + pub fn set_status_bits(&self, bits: u16) { + self.regs.lock().unwrap()[1] |= ((bits & self.status_w1c_mask) as u32) << 16; + } + + /// Set the board-assigned Subsystem ID, which survives function reset. + pub fn set_subsystem_identity(&self, value: u32) { + self.regs.lock().unwrap()[11] = value; + } + + /// The base address firmware programmed into BAR `i`, type bits masked. + pub fn bar_base(&self, i: usize) -> u32 { + let r = self.regs.lock().unwrap(); + let v = r[4 + i]; + match self.bars[i] { + BarSlot::Bar(Bar::Io(_)) => v & !3, + // Above 4 GiB: out of reach of a 32-bit PCI address. + BarSlot::Bar(Bar::Mem64(_)) if r[5 + i] != 0 => 0, + _ => v & !15, + } + } + + pub fn command(&self) -> u32 { + self.regs.lock().unwrap()[1] & 0xffff + } + + pub fn interrupt_pin(&self) -> u8 { + (self.regs.lock().unwrap()[15] >> 8) as u8 + } + + /// The regions this function decodes: (space, BAR, base, size) for + /// each BAR that has been assigned an address in a space the command + /// register enables, in BAR order. The bus caches these and updates its + /// mappings after a configuration write. + pub fn bar_regions(&self) -> Vec<(Space, usize, u32, u32)> { + let cmd = self.command(); + (0..6) + .filter_map(|i| { + let (space, size) = match self.bars[i] { + BarSlot::Bar(Bar::Io(n)) if cmd & PCI_COMMAND_IO != 0 => (Space::Io, n), + BarSlot::Bar(Bar::Mem(n) | Bar::Mem64(n)) if cmd & PCI_COMMAND_MEMORY != 0 => (Space::Mem, n), + _ => return None, + }; + let base = self.bar_base(i); + (base != 0).then_some((space, i, base, size)) + }) + .collect() + } +} + +/// The header's registers. Identity, class and the BARs' sizes come from +/// the function, so a header only loads into the same kind of function. +impl crate::traits::Saveable for PciConfig { + fn save_state(&self) -> toml::Value { + let mut t = toml::map::Map::new(); + t.insert("regs".into(), u32_slice_to_toml(&*self.regs.lock().unwrap())); + toml::Value::Table(t) + } + + fn load_state(&self, v: &toml::Value) -> Result<(), String> { + let Some(r) = get_field(v, "regs") else { return Ok(()) }; + let mut saved = [0u32; 64]; + load_u32_slice(r, &mut saved); + let mut regs = self.regs.lock().unwrap(); + if saved[0] != regs[0] { + return Err(format!("pci: saved header is for {:#010x}, this function is {:#010x}", saved[0], regs[0])); + } + *regs = saved; + Ok(()) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + fn scsi_like() -> PciConfig { + PciConfig::new(0x9004, 0x8078, 0x0100_00, 1, + [Bar::Io(0x100), Bar::Mem(0x1000), Bar::Unused, Bar::Unused, Bar::Unused, Bar::Unused], PCI_INTA) + } + + fn scsi_like_with_policy(policy: PciConfigPolicy<'_>) -> PciConfig { + PciConfig::new_with_policy( + 0x9004, 0x8078, 0x0100_00, 1, + [Bar::Io(0x100), Bar::Mem(0x1000), Bar::Unused, Bar::Unused, Bar::Unused, Bar::Unused], PCI_INTA, + policy, + ) + } + + /// Firmware learns how much space a device wants by writing all ones to + /// a BAR and reading back which address bits stuck. + #[test] + fn save_load_round_trip() { + use crate::traits::Saveable; + let src = scsi_like(); + src.write(1, PCI_COMMAND_MEMORY | PCI_COMMAND_IO); + src.write(4, 0x0000_1000); + src.write(5, 0x8000_1000); + src.write(15, 0x0000_0108); + let v1 = src.save_state(); + let dst = scsi_like(); + dst.load_state(&v1).expect("load_state"); + assert_eq!(v1, dst.save_state(), "PciConfig save_state mismatch after load_state round-trip"); + assert_eq!(dst.bar_base(1), 0x8000_1000); + let other = PciConfig::new(0x1234, 0x5678, 0, 0, [Bar::Unused; 6], 0); + assert!(other.load_state(&v1).is_err(), "a header loads only into the same kind of function"); + } + + fn nvme_like() -> PciConfig { + PciConfig::new(0x1b36, 0x0010, 0x01_0802, 1, + [Bar::Mem64(0x2000), Bar::Unused, Bar::Unused, Bar::Unused, Bar::Unused, Bar::Unused], PCI_INTA) + } + + /// A 64-bit BAR reads as memory type 10b, sizes through its lower dword, + /// and keeps whatever its upper dword is given. + #[test] + fn a_64_bit_bar_probes_and_stores_both_halves() { + let c = nvme_like(); + assert_eq!(c.read(4), 0x4, "64-bit memory, not prefetchable"); + c.write(4, 0xffff_ffff); + c.write(5, 0xffff_ffff); + assert_eq!(c.read(4), 0xffff_e004, "8 KiB, type bits kept"); + assert_eq!(c.read(5), 0xffff_ffff, "the upper half is plain storage"); + c.write(5, 0); + c.write(4, 0x8000_2000); + assert_eq!(c.bar_base(0), 0x8000_2000); + assert_eq!(c.read(4), 0x8000_2004); + c.write(6, 0xffff_ffff); + assert_eq!(c.read(6), 0, "the BAR after the pair is its own"); + } + + #[test] + fn a_64_bit_bar_decodes_below_4_gib_only() { + let c = nvme_like(); + c.write(4, 0x8000_2000); + c.write(1, PCI_COMMAND_MEMORY); + assert_eq!(c.bar_regions(), vec![(Space::Mem, 0, 0x8000_2000, 0x2000)]); + c.write(5, 1); + assert!(c.bar_regions().is_empty(), "a base above 4 GiB cannot be reached"); + c.write(1, 0); + c.reset(); + assert_eq!((c.read(4), c.read(5)), (4, 0), "reset forgets both halves"); + } + + #[test] + fn a_64_bit_bar_saves_with_its_upper_half() { + use crate::traits::Saveable; + let src = nvme_like(); + src.write(4, 0x8000_2000); + src.write(5, 3); + let dst = nvme_like(); + dst.load_state(&src.save_state()).unwrap(); + assert_eq!((dst.read(4), dst.read(5)), (0x8000_2004, 3)); + } + + #[test] + fn a_bar_reports_its_size_to_a_probe() { + let c = scsi_like(); + assert_eq!(c.read(0), 0x8078_9004, "device and vendor id"); + c.write(4, 0xffff_ffff); + assert_eq!(c.read(4), 0xffff_ff01, "256 bytes of I/O space"); + c.write(5, 0xffff_ffff); + assert_eq!(c.read(5), 0xffff_f000, "4 KiB of memory space"); + // An unimplemented BAR reads back as zero however hard it is probed. + c.write(6, 0xffff_ffff); + assert_eq!(c.read(6), 0); + c.write(5, 0x8000_1000); + assert_eq!(c.bar_base(1), 0x8000_1000, "and then holds the address assigned to it"); + } + + /// No function has an expansion ROM, so the ROM base register is + /// unimplemented: a size probe reads back zero, which tells firmware + /// there is no ROM to map (PCI 2.2, 6.1 and 6.2.5.2). + #[test] + fn the_expansion_rom_register_reads_zero_when_probed() { + let c = scsi_like(); + c.write(12, 0xffff_f800); + assert_eq!(c.read(12), 0); + c.write(12, 0x00b0_2001); + assert_eq!(c.read(12), 0, "an assigned address does not stick either"); + } + + /// A function decodes a space only once it has been told it may, and + /// only the BARs that have been assigned an address. + #[test] + fn a_function_claims_only_its_own_region_and_only_when_enabled() { + let c = scsi_like(); + c.write(5, 0x8000_1000); + assert!(c.bar_regions().is_empty(), "memory decoding is off until enabled"); + c.write(1, PCI_COMMAND_MEMORY); + assert_eq!(c.bar_regions(), vec![(Space::Mem, 1, 0x8000_1000, 0x1000)], "BAR 1 alone, and not the I/O BAR"); + } + + #[test] + fn reset_returns_the_header_to_power_up() { + let c = scsi_like(); + let fresh: Vec = (0..64).map(|i| c.read(i)).collect(); + c.write(1, PCI_COMMAND_MEMORY | PCI_COMMAND_IO); + c.write(4, 0x0000_1000); + c.write(5, 0x8000_1000); + c.write(15, 0x0000_0009); + c.reset(); + assert_eq!((0..64).map(|i| c.read(i)).collect::>(), fresh); + assert!(c.bar_regions().is_empty()); + assert_eq!(c.interrupt_pin(), PCI_INTA, "the pin is wiring, not state"); + } + + /// A subsystem identity set after construction, as IOC3 sets its own, + /// is the board's and survives a reset. + #[test] + fn reset_keeps_the_subsystem_identity() { + let c = scsi_like(); + c.set_subsystem_identity(0xc304_10a9); + c.write(4, 0x0000_1000); + c.reset(); + assert_eq!(c.read(11), 0xc304_10a9); + assert_eq!(c.read(4), 1); + } + + /// Firmware programs the latency timer and cache line size, and reads + /// them back; the header type byte above them stays what it is. + #[test] + fn the_latency_timer_and_cache_line_size_are_stored() { + let c = scsi_like(); + c.write(3, 0x00ff_2810); + assert_eq!(c.read(3), 0x0000_2810); + c.reset(); + assert_eq!(c.read(3), 0); + } + + #[test] + fn the_interrupt_pin_is_read_only() { + let c = scsi_like(); + c.write(15, 0x0000_0009); + assert_eq!(c.read(15) & 0xffff, 0x0109, "line 9 assigned, pin A kept"); + c.write(15, 0); + assert_eq!(c.interrupt_pin(), PCI_INTA); + } + + #[test] + fn header_policies_mask_writes_and_restore_reset_values() { + for (status, command, latency, cache) in [(0x0200, 0xffff, 0, 0), (0x02b0, 0x0156, 0xf0, 0x0f)] { + let c = scsi_like_with_policy(PciConfigPolicy { + status, + status_w1c_mask: 0, + command_mask: command, + latency_mask: latency, + cache_line_mask: cache, + registers: &[PciRegisterPolicy { + reg_dword: 18, + initial: 0xaabb_0020, + writable_mask: 0xff00, + }], + }); + c.write(1, u32::MAX); + c.write(3, u32::MAX); + c.write(18, u32::MAX); + assert_eq!(c.read(1), (u32::from(status) << 16) | u32::from(command)); + assert_eq!(c.read(3), (u32::from(latency) << 8) | u32::from(cache)); + assert_eq!(c.read(18), 0xaabb_ff20); + c.reset(); + assert_eq!(c.read(1), u32::from(status) << 16); + assert_eq!(c.read(3), 0); + assert_eq!(c.read(18), 0xaabb_0020); + } + } + + #[test] + fn masked_writes_preserve_unselected_fields_and_status_events() { + let c = scsi_like_with_policy(PciConfigPolicy { + status: 0x0200, + status_w1c_mask: 0x8040, + command_mask: 0x0157, + latency_mask: 0xff, + cache_line_mask: 0xff, + registers: &[PciRegisterPolicy { + reg_dword: 18, + initial: 0xaabb_0000, + writable_mask: 0xffff, + }], + }); + c.set_status_bits(0x8040); + c.write(1, 0x0157); + c.write(3, 0x1234); + c.write(5, 0x1234_5000); + c.write(4, 0x1201); + for (reg, value, mask, expected) in [ + (1, 0xffff_0002, 0xff, 0x8240_0102), + (1, 0xffff_0000, 0x8000_0000, 0x0240_0102), + (1, 0x0040_0000, 0x00ff_0000, 0x0200_0102), + (1, u32::MAX, 0, 0x0200_0102), + (3, 0xffff_00ab, 0xff, 0x12ab), + (5, 0xabcd_ffff, 0xffff_0000, 0xabcd_5000), + (4, 0x3400, 0xff00, 0x3401), + (15, u32::MAX, 0xff00, 0x0100), + (18, u32::MAX, 0xff, 0xaabb_00ff), + ] { + c.write_masked(reg, value, mask); + assert_eq!(c.read(reg), expected, "register {reg}"); + } + c.set_status_bits(u16::MAX); + c.reset(); + assert_eq!(c.read(1), 0x0200_0000); + let default = scsi_like(); + default.set_status_bits(u16::MAX); + assert_eq!(default.read(1), 0x0200_0000); + } + + #[test] + fn snapshots_restore_registers_with_destination_write_policies() { + use crate::traits::Saveable; + let src = scsi_like_with_policy(PciConfigPolicy { + status: 0x02b0, + status_w1c_mask: 0xc000, + command_mask: 0x0157, + latency_mask: 0xff, + cache_line_mask: 0xff, + registers: &[PciRegisterPolicy { + reg_dword: 18, + initial: 0, + writable_mask: 0x100, + }], + }); + src.set_status_bits(0x8000); + src.write(3, 0x1234); + src.write(18, 0x100); + let saved = src.save_state(); + for (events, command, cache) in [(0, 7, 0x0f), (0xc000, 0x0157, 0xff)] { + let dst = scsi_like_with_policy(PciConfigPolicy { + status: 0x0200, + status_w1c_mask: events, + command_mask: command, + latency_mask: 0, + cache_line_mask: cache, + registers: &[PciRegisterPolicy { + reg_dword: 18, + initial: 0x20, + writable_mask: 0x200, + }], + }); + dst.load_state(&saved).unwrap(); + assert_eq!(dst.save_state(), saved); + assert_eq!(dst.read(1), 0x0200_0000 | ((0x8000 & u32::from(events)) << 16)); + dst.write_masked(1, u32::MAX, u32::MAX); + dst.write(3, u32::MAX); + dst.write(18, u32::MAX); + assert_eq!(dst.read(1), 0x0200_0000 | u32::from(command)); + assert_eq!(dst.read(3), (0x1234 & !u32::from(cache)) | u32::from(cache)); + assert_eq!(dst.read(18), 0x300); + dst.reset(); + assert_eq!(dst.read(18), 0x20); + } + } +} diff --git a/src/pci/decode.rs b/src/pci/decode.rs new file mode 100644 index 000000000..b813c6100 --- /dev/null +++ b/src/pci/decode.rs @@ -0,0 +1,472 @@ +//! What a bus decodes: the functions placed on it, the BAR regions their +//! headers have programmed, and the runs of a host's CPU windows that one +//! BAR covers whole. +//! +//! The function table is write-once, so a lookup returns a borrow. The BAR +//! regions are a table of packed words that a CPU access scans without a +//! lock: a function's BAR comparators are its own registers on the bus, not +//! anything the bus locks, and configuration writes and window accesses +//! both come from the CPU. Whoever changes a header or a host window +//! updates the mappings under the publish mutex, which also holds the host's +//! windows as of the last update; `make` runs inside it, so two CPUs +//! writing registers cannot publish out of order. +//! +//! A bus with PCI-to-PCI bridges decodes for the buses behind them too: the +//! regions of a secondary bus's functions that the bridge's header forwards +//! are merged into this bus's table, so one scan, and the runs the host maps, +//! reach a function however many bridges deep it sits. A function behind a +//! bridge has a slot in `fwd`, which only grows, so a lookup still returns a +//! borrow. + +use std::sync::atomic::{AtomicU64, AtomicUsize, Ordering}; +use std::sync::{Arc, Mutex, OnceLock}; + +use super::{DevFn, PciFunction, Space}; + +/// A host window onto PCI: CPU addresses `cpu..cpu + len` reach PCI `space` +/// from address `pci`. `mask` is the CPU address bits the host decodes, so +/// the window repeats at every address that agrees on them (MACE: all 32; +/// the Bridge: its widget window bits). The host, not the bus, chooses each +/// window's byte order. +#[derive(Clone, Copy, Debug)] +pub struct CpuWindow { + pub space: Space, + pub cpu: u64, + pub pci: u32, + pub len: u64, + pub mask: u64, +} + +/// A run of CPU addresses, in whole decode-table pages, that one BAR covers. +/// The host maps it on the physical bus so the processor reaches the function +/// with no search of the bus. `off` is the BAR offset of the run's first +/// byte, and `window` is the index of the [`CpuWindow`] the run lies in. +#[derive(Clone)] +pub struct BarMapping { + pub cpu: u64, + pub len: u64, + pub function: Arc, + pub bar: usize, + pub off: u32, + pub window: u16, +} + +/// Where a bus's runs go: the whole table, on every publish. The host maps +/// each run over its window and gives the pages the last call named back. +pub type BarMappingSink = Box) + Send + Sync>; + +/// The most BAR regions open at once on a bus; more are ignored with a +/// warning. +const MAX_REGIONS: usize = 64; + +/// Functions behind bridges that one bus can decode for. +const MAX_FORWARDED: usize = 256; + +/// One BAR region of one function: the range BAR `bar` of `function` +/// decodes in `space`. +#[derive(Clone)] +pub(super) struct BarRegion { + pub space: Space, + pub bar: usize, + pub base: u32, + pub size: u32, + pub function: Arc, +} + + +/// Functions per device, and so the function table's stride. +const FUNCTIONS: usize = 8; + +/// A BAR region as the slow table holds it, packed into one word: base +/// (bits 0-31), log2 of the size (32-37), I/O space (38), function slot +/// (40-51: `device * 8 + function`, or 256 and up for a function behind a +/// bridge) and BAR (52-54). +#[derive(Clone, Copy, PartialEq, Eq, Debug)] +struct PackedRegion { + space: Space, + func: u16, + bar: usize, + base: u32, + size: u32, +} + +impl PackedRegion { + fn pack(self) -> u64 { + self.base as u64 + | (self.size.trailing_zeros() as u64) << 32 + | ((self.space == Space::Io) as u64) << 38 + | (self.func as u64) << 40 + | (self.bar as u64) << 52 + } + + fn unpack(w: u64) -> Self { + Self { + space: if w & (1 << 38) != 0 { Space::Io } else { Space::Mem }, + func: ((w >> 40) & 0xfff) as u16, + bar: ((w >> 52) & 7) as usize, + base: w as u32, + size: 1u32 << ((w >> 32) & 0x3f), + } + } +} + +/// The publish mutex's contents. +#[derive(Default)] +struct Publish { + windows: Vec, + sink: Option, +} + +pub(super) struct Decode { + /// By `device * 8 + function`; set once. + funcs: Box<[OnceLock>]>, + /// Functions behind bridges that have had a region, in the order they + /// first did; append-only, so an index never changes meaning. + fwd: Box<[OnceLock>]>, + nfwd: AtomicUsize, + /// The slow table: first match wins, in (device, function, BAR) order. + regions: [AtomicU64; MAX_REGIONS], + count: AtomicUsize, + publish: Mutex, +} + +impl Decode { + pub(super) fn new(devices: u8) -> Self { + Self { + funcs: (0..devices as usize * FUNCTIONS).map(|_| OnceLock::new()).collect(), + fwd: (0..MAX_FORWARDED).map(|_| OnceLock::new()).collect(), + nfwd: AtomicUsize::new(0), + regions: std::array::from_fn(|_| AtomicU64::new(0)), + count: AtomicUsize::new(0), + publish: Mutex::new(Publish::default()), + } + } + + /// Device numbers on the bus. + pub(super) fn devices(&self) -> u8 { + (self.funcs.len() / FUNCTIONS) as u8 + } + + fn index(&self, at: DevFn) -> Option { + let i = at.device as usize * FUNCTIONS + at.function as usize; + (at.function < FUNCTIONS as u8 && i < self.funcs.len()).then_some(i) + } + + /// Place a function. False if the address is out of range or taken. + pub(super) fn insert(&self, at: DevFn, f: Arc) -> bool { + self.index(at).is_some_and(|i| self.funcs[i].set(f).is_ok()) + } + + pub(super) fn function(&self, at: DevFn) -> Option<&Arc> { + self.funcs.get(self.index(at)?)?.get() + } + + /// Every function, in (device, function) order. + pub(super) fn functions(&self) -> impl Iterator)> { + self.funcs.iter().enumerate().filter_map(|(i, f)| { + Some((DevFn { device: (i / FUNCTIONS) as u8, function: (i % FUNCTIONS) as u8 }, f.get()?)) + }) + } + + /// The regions of this bus's functions and of the buses behind its + /// bridges, as forwarded. No lock: the function table is write-once. + pub(super) fn all_regions(&self) -> Vec { + self.regions().into_iter().map(|(w, _)| w).collect() + } + + /// Own regions first, each with its slot in `funcs`; then, for each + /// bridge, the regions behind it that its header forwards (with no slot + /// yet). + fn regions(&self) -> Vec<(BarRegion, Option)> { + let mut out = Vec::new(); + for (at, f) in self.functions() { + let slot = (at.device as usize * FUNCTIONS + at.function as usize) as u16; + for (space, bar, base, size) in f.state().config().bar_regions() { + out.push((BarRegion { space, bar, base, size, function: f.clone() }, Some(slot))); + } + if let Some(bus) = f.secondary() { + out.extend(bus.all_regions().into_iter().filter(|r| f.state().config().forwards(r.space, r.base, r.size)).map(|r| (r, None))); + } + } + out + } + + /// The slot in `fwd` of a function behind a bridge, given one if it has + /// none. None once the table is full. + fn forwarded_slot(&self, f: &Arc) -> Option { + let n = self.nfwd.load(Ordering::Acquire); + if let Some(i) = self.fwd[..n].iter().position(|s| s.get().is_some_and(|g| Arc::ptr_eq(g, f))) { + return Some((FUNCTIONS * 32 + i) as u16); + } + let slot = self.fwd.get(n)?; + let _ = slot.set(f.clone()); + self.nfwd.store(n + 1, Ordering::Release); + Some((FUNCTIONS * 32 + n) as u16) + } + + /// Where the runs go. The first publish happens now. + pub(super) fn set_mapping_sink(&self, sink: BarMappingSink) { + let mut p = self.publish.lock().unwrap(); + p.sink = Some(sink); + self.build(&p); + } + + /// Rebuild the slow table and publish the runs. With `make`, the host's + /// windows are replaced first (inside the publish mutex, so the host's + /// own lock nests under it); without it, the last windows are kept, which + /// is the call for a header change. + pub(super) fn update_mappings(&self, make: Option<&dyn Fn() -> Vec>) { + let mut p = self.publish.lock().unwrap(); + if let Some(make) = make { + p.windows = make(); + } + self.build(&p); + } + + /// The function, BAR and BAR offset that `addr` reaches in `space`. + pub(super) fn decode(&self, space: Space, addr: u32) -> Option<(&dyn PciFunction, usize, u32)> { + let n = self.count.load(Ordering::Acquire); + self.regions[..n].iter().find_map(|w| { + let w = PackedRegion::unpack(w.load(Ordering::Relaxed)); + let off = addr.wrapping_sub(w.base); + if w.space != space || off >= w.size { + return None; + } + let f = match (w.func as usize).checked_sub(FUNCTIONS * 32) { + None => self.funcs[w.func as usize].get()?, + Some(i) => self.fwd[i].get()?, + }; + Some((&**f, w.bar, off)) + }) + } + + fn build(&self, p: &Publish) { + let wins = self.regions(); + let mut n = 0; + for (BarRegion { space, bar, base, size, function: f }, slot) in &wins { + if n == MAX_REGIONS { + log::warn!("pci: more than {MAX_REGIONS} BAR regions open; one of BAR {bar} ignored"); + continue; + } + let Some(func) = slot.or_else(|| self.forwarded_slot(f)) else { + log::warn!("pci: more than {MAX_FORWARDED} functions behind bridges; BAR {bar} ignored"); + continue; + }; + self.regions[n].store(PackedRegion { space: *space, func, bar: *bar, base: *base, size: *size }.pack(), Ordering::Relaxed); + n += 1; + } + self.count.store(n, Ordering::Release); + if let Some(sink) = &p.sink { + sink(Self::mappings(&wins, &p.windows)); + } + } + + /// The pages of each host window that one BAR decodes whole. Pages a + /// BAR covers only in part, and pages two BARs claim (a misconfiguration + /// the host resolves its own way), are left to the host. + fn mappings(wins: &[(BarRegion, Option)], windows: &[CpuWindow]) -> Vec { + const PAGE: u64 = 1 << crate::ppmem::devmap::SLOT_SHIFT; + let mut runs = Vec::new(); + for (wi, w) in windows.iter().enumerate() { + let (wlo, whi) = (w.pci as u64, w.pci as u64 + w.len); + // (first page, end page, function, BAR, BAR base) in PCI addresses. + let mut found: Vec<(u64, u64, &Arc, usize, u64)> = Vec::new(); + for (BarRegion { space, bar, base, size, function: f }, _) in wins { + if *space != w.space { + continue; + } + let (lo, hi) = ((*base as u64).max(wlo), (*base as u64 + *size as u64).min(whi)); + let (lo, hi) = (lo.next_multiple_of(PAGE), hi & !(PAGE - 1)); + if lo < hi { + found.push((lo, hi, f, *bar, *base as u64)); + } + } + for (i, &(lo, hi, f, bar, base)) in found.iter().enumerate() { + if found.iter().enumerate().any(|(j, &(l, h, ..))| j != i && l < hi && lo < h) { + continue; + } + runs.push(BarMapping { + cpu: w.cpu.wrapping_add(lo).wrapping_sub(wlo), + len: hi - lo, + function: f.clone(), + bar, + off: (lo - base) as u32, + window: wi as u16, + }); + } + } + runs + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::pci::testing::TestCard; + use crate::pci::{Bar, PCI_COMMAND_IO, PCI_COMMAND_MEMORY}; + + const PAGE: u32 = 0x1000; + + fn at(device: u8, function: u8) -> DevFn { + DevFn { device, function } + } + + /// An I/O window and a memory window, as the O2 has them. + fn windows() -> Vec { + let w = |space, cpu, pci, len| CpuWindow { space, cpu, pci, len, mask: 0xffff_ffff }; + vec![w(Space::Io, 0x1800_0000, 0, 0x200_0000), w(Space::Mem, 0x1a00_0000, 0x8000_0000, 0x200_0000)] + } + + struct Rig { + d: Decode, + runs: Arc>>, + cards: Vec>, + } + + /// A table with one card per BAR layout, its memory BARs placed one after + /// another from the base named; memory decoding is on. + fn rig(cards: &[([Bar; 6], u32)]) -> Rig { + let d = Decode::new(32); + let mut made = Vec::new(); + for (i, (bars, base)) in cards.iter().enumerate() { + let c = TestCard::new(*bars, 0); + let mut next = *base; + for (n, b) in bars.iter().enumerate() { + if let Bar::Mem(size) = b { + c.state().config().write(4 + n, next); + next += size; + } + } + c.state().config().write(1, PCI_COMMAND_MEMORY | PCI_COMMAND_IO); + assert!(d.insert(at(i as u8 + 1, 0), c.clone())); + made.push(c); + } + let runs = Arc::new(Mutex::new(Vec::new())); + let sink = runs.clone(); + d.set_mapping_sink(Box::new(move |r| *sink.lock().unwrap() = r)); + d.update_mappings(Some(&windows)); + Rig { d, runs, cards: made } + } + + fn spans(r: &Rig) -> Vec<(u64, u64, usize, u32, u16)> { + r.runs.lock().unwrap().iter().map(|b| (b.cpu, b.len, b.bar, b.off, b.window)).collect() + } + + fn mem(n: u32) -> [Bar; 6] { + [Bar::Unused, Bar::Mem(n), Bar::Unused, Bar::Unused, Bar::Unused, Bar::Unused] + } + + #[test] + fn a_whole_page_bar_gives_one_run_with_its_window_index() { + let r = rig(&[(mem(2 * PAGE), 0x8000_4000)]); + assert_eq!(spans(&r), vec![(0x1a00_4000, 0x2000, 1, 0, 1)]); + } + + #[test] + fn a_bar_larger_than_the_window_starts_part_way_in() { + // BAR 0x8000_0000..0x8000_4000 is cut by a window that starts at + // 0x8000_2000, so the run's first byte is BAR offset 0x2000. + let r = rig(&[(mem(4 * PAGE), 0x8000_0000)]); + r.d.update_mappings(Some(&|| vec![CpuWindow { space: Space::Mem, cpu: 0x1a00_0000, pci: 0x8000_2000, len: 0x1_0000, mask: 0xffff_ffff }])); + assert_eq!(spans(&r), vec![(0x1a00_0000, 0x2000, 1, 0x2000, 0)]); + } + + #[test] + fn a_bar_move_updates_the_mappings() { + let r = rig(&[(mem(PAGE), 0x8000_1000)]); + assert_eq!(spans(&r)[0].0, 0x1a00_1000); + r.cards[0].state().config().write(5, 0x8000_3000); + r.d.update_mappings(None); + assert_eq!(spans(&r), vec![(0x1a00_3000, 0x1000, 1, 0, 1)]); + assert!(r.d.decode(Space::Mem, 0x8000_1000).is_none() && r.d.decode(Space::Mem, 0x8000_3004).is_some()); + } + + #[test] + fn a_command_register_disable_removes_the_run() { + let r = rig(&[(mem(PAGE), 0x8000_1000)]); + r.cards[0].state().config().write(1, PCI_COMMAND_IO); + r.d.update_mappings(None); + assert!(spans(&r).is_empty()); + assert!(r.d.decode(Space::Mem, 0x8000_1000).is_none()); + } + + #[test] + fn a_sub_page_bar_has_no_run_but_bar_access_finds_it() { + let r = rig(&[(mem(0x100), 0x8000_1000)]); + assert!(spans(&r).is_empty()); + let (_, bar, off) = r.d.decode(Space::Mem, 0x8000_1084).unwrap(); + assert_eq!((bar, off), (1, 0x84)); + assert!(r.d.decode(Space::Mem, 0x8000_1100).is_none(), "past the end"); + } + + #[test] + fn two_bars_on_one_page_have_no_run_and_first_match_wins_in_bar_access() { + let r = rig(&[(mem(PAGE), 0x8000_1000), (mem(PAGE), 0x8000_1000)]); + assert!(spans(&r).is_empty()); + let (f, ..) = r.d.decode(Space::Mem, 0x8000_1000).unwrap(); + assert!(std::ptr::eq(f as *const dyn PciFunction as *const u8, &*r.cards[0] as *const TestCard as *const u8)); + } + + /// Ten cards with six one-page BARs each is 60 regions; an eleventh + /// passes the slow table's 64. Runs are published in (device, BAR) order. + #[test] + fn a_table_of_64_runs_is_published_in_order() { + let cards: Vec<_> = (0..11).map(|i| ([Bar::Mem(PAGE); 6], 0x8000_0000 + i * 6 * PAGE)).collect(); + let r = rig(&cards[..10]); + let got = spans(&r); + assert_eq!(got.len(), 60); + let cpus: Vec = got.iter().map(|g| g.0).collect(); + let mut sorted = cpus.clone(); + sorted.sort(); + assert_eq!(cpus, sorted, "device then BAR order is address order here"); + // Past the table the extra regions are dropped from decode with a + // warning, but their runs still publish: the run table has no limit. + let r = rig(&cards); + assert_eq!(spans(&r).len(), 66); + assert!(r.d.decode(Space::Mem, 0x8000_0000 + 65 * PAGE).is_none(), "the 65th region is not in the slow table"); + } + + #[test] + fn runs_agree_with_bar_access() { + let r = rig(&[(mem(2 * PAGE), 0x8000_4000), (mem(PAGE), 0x8000_8000)]); + for run in r.runs.lock().unwrap().iter() { + let w = windows()[run.window as usize]; + for page in 0..run.len / PAGE as u64 { + let pci = (w.pci as u64 + (run.cpu - w.cpu) + page * PAGE as u64) as u32; + let (f, bar, off) = r.d.decode(w.space, pci).expect("a run's page is decoded"); + assert!(std::ptr::eq(f as *const dyn PciFunction as *const u8, &*run.function as *const dyn PciFunction as *const u8)); + assert_eq!((bar, off), (run.bar, run.off + page as u32 * PAGE)); + } + } + } + + #[test] + fn two_threads_updating_the_mappings_end_with_the_later_makes_result() { + let r = Arc::new(rig(&[(mem(PAGE), 0x8000_1000)])); + let at_cpu = |cpu: u64| move || vec![CpuWindow { space: Space::Mem, cpu, pci: 0x8000_0000, len: 0x1_0000, mask: 0xffff_ffff }]; + let (tx, rx) = std::sync::mpsc::channel::<()>(); + let (go_tx, go_rx) = std::sync::mpsc::channel::<()>(); + let first = { + let r = r.clone(); + let make = at_cpu(0x1000_0000); + std::thread::spawn(move || { + r.d.update_mappings(Some(&|| { + tx.send(()).unwrap(); + go_rx.recv().unwrap(); + make() + })) + }) + }; + rx.recv().unwrap(); + let second = { + let r = r.clone(); + std::thread::spawn(move || r.d.update_mappings(Some(&at_cpu(0x2000_0000)))) + }; + // The second call waits on the publish mutex; let the first finish. + std::thread::sleep(std::time::Duration::from_millis(50)); + go_tx.send(()).unwrap(); + first.join().unwrap(); + second.join().unwrap(); + assert_eq!(spans(&r)[0].0, 0x2000_1000, "the later make is what the sink last saw"); + } +} diff --git a/src/pci/function.rs b/src/pci/function.rs new file mode 100644 index 000000000..d239d1e62 --- /dev/null +++ b/src/pci/function.rs @@ -0,0 +1,176 @@ +//! A function as the bus sees it: the trait a function implements, the +//! state the bus fills in when it attaches one, where it sits, and how a +//! bus master reaches memory. + +use std::sync::{Arc, OnceLock}; + +use crate::irq::IrqLine; +use super::{PciBus, PciConfig}; + +/// What the bus sees of one function: its configuration header, and the +/// interrupt line and the way to memory the bus gives it when it is +/// attached, each set once. A function that raises no interrupt (RAD1) +/// never has a line. +pub struct PciFunctionState { + config: PciConfig, + irq: OnceLock, + dma: OnceLock>, +} + +impl PciFunctionState { + pub fn new(config: PciConfig) -> Self { + Self { config, irq: OnceLock::new(), dma: OnceLock::new() } + } + + pub fn config(&self) -> &PciConfig { + &self.config + } + + pub fn irq(&self) -> Option<&IrqLine> { + self.irq.get() + } + + pub fn dma(&self) -> Option<&Arc> { + self.dma.get() + } + + /// The bus wires the function's interrupt pin to `line`. Only the first + /// line counts. + pub fn connect_irq(&self, line: IrqLine) { + let _ = self.irq.set(line); + } + + /// The bus gives a function that masters the bus its way to memory. + /// Only the first counts. + pub fn connect_dma(&self, dma: Arc) { + let _ = self.dma.set(dma); + } +} + +/// A PCI function, as a host bridge sees it. +/// +/// BAR accesses arrive with the offset and value the device sees: whatever +/// the bridge does to byte order it has already done. +pub trait PciFunction: Send + Sync { + /// The function's header, and the interrupt line and DMA the bus gave it. + fn state(&self) -> &PciFunctionState; + + /// What a configuration read of dword `index` returns to the guest. The + /// header as stored, unless the chip decodes fewer address lines than + /// the bus gives it (IOC3 repeats its first 32 bytes). + fn config_dword(&self, index: usize) -> u32 { + self.state().config().read(index) + } + + /// A read of `width` bytes (1, 2 or 4) at `off` within BAR `bar`. + fn bar_read(&self, bar: usize, off: u32, width: usize) -> u32; + + fn bar_write(&self, bar: usize, off: u32, val: u32, width: usize); + + /// The bus has attached the function and filled in its state's + /// interrupt line and DMA. A function that builds on either does it + /// here: IOC3 wraps its DMA and drives its pin, virtio hands both to its + /// shared core, and the SCSI controllers drive their pins. + fn attached(&self) {} + + /// The bus wrote configuration dword `reg` of this function's header and + /// it changed from `old` to `new`. A PCI-to-PCI bridge acts on bridge + /// control here; every other function has nothing to do. + fn config_written(&self, _reg: usize, _old: u32, _new: u32) {} + + /// The bus behind this function, if it is a PCI-to-PCI bridge. + fn secondary(&self) -> Option<&PciBus> { + None + } + + /// RST# reaches the function after the bus has reset its header: drop + /// the interrupt line and return to power-on state. A function never + /// resets its own header, so the bus's decode table cannot go stale. + fn reset(&self) {} +} + +/// Where a function sits on a bus: its device number and its function number. +/// Orders by device, then function, the order RST# visits them in. +#[derive(Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash, Debug)] +pub struct DevFn { + pub device: u8, + pub function: u8, +} + +/// How a bus master's access ended. +#[must_use] +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum PciDmaStatus { + Ok, + /// Some of the bytes had no target: reads return zero for them and + /// writes drop them. A bridge may also log it and raise its error + /// interrupt. + MasterAbort, +} + +impl PciDmaStatus { + pub fn is_ok(self) -> bool { + self == PciDmaStatus::Ok + } +} + +/// Memory as a bus master reaches it. The bridge decides what a PCI address +/// is and in which order its bytes arrive; the function sees bytes at bus +/// addresses, as it would on the bus. +/// +/// A function calls only the object the bus left in its state and never +/// learns whether it is shared. The O2's MACE hands every function one +/// object (its translation is fixed); a bridge whose translation depends on the +/// mastering device, as the Octane's does, hands each its own. +pub trait PciDma: Send + Sync { + /// `buf.len()` bytes from bus address `addr`, 64 bits wide for masters + /// that use dual address cycles. + fn dma_read(&self, addr: u64, buf: &mut [u8]) -> PciDmaStatus; + + /// `data` to bus address `addr`. + fn dma_write(&self, addr: u64, data: &[u8]) -> PciDmaStatus; +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::pci::{Bar, PCI_INTA}; + + struct Func(PciFunctionState); + impl PciFunction for Func { + fn state(&self) -> &PciFunctionState { &self.0 } + fn bar_read(&self, _: usize, off: u32, _: usize) -> u32 { off } + fn bar_write(&self, _: usize, _: u32, _: u32, _: usize) {} + } + + fn func() -> Func { + Func(PciFunctionState::new(PciConfig::new(0x9004, 0x8078, 0x0100_00, 1, [Bar::Unused; 6], PCI_INTA))) + } + + #[test] + fn a_state_takes_its_line_and_dma_once() { + struct Dma; + impl PciDma for Dma { + fn dma_read(&self, _: u64, _: &mut [u8]) -> PciDmaStatus { PciDmaStatus::Ok } + fn dma_write(&self, _: u64, _: &[u8]) -> PciDmaStatus { PciDmaStatus::Ok } + } + let (a, b) = (crate::irq::LevelProbe::line(), crate::irq::LevelProbe::line()); + let f = func(); + assert!(f.state().irq().is_none() && f.state().dma().is_none()); + f.state().connect_irq(a.1); + f.state().connect_irq(b.1); + f.state().connect_dma(Arc::new(Dma)); + f.state().irq().unwrap().set(true); + assert!(a.0.level() && !b.0.level(), "the second line is ignored"); + assert!(f.state().dma().is_some()); + assert!(func().state().irq().is_none(), "a function with no interrupt never has a line"); + } + + #[test] + fn devfn_orders_by_device_then_function() { + let a = |device, function| DevFn { device, function }; + let mut v = [a(3, 0), a(1, 1), a(1, 0)]; + v.sort(); + assert_eq!(v, [a(1, 0), a(1, 1), a(3, 0)]); + } +} diff --git a/src/pci/host.rs b/src/pci/host.rs new file mode 100644 index 000000000..2f28fbb5d --- /dev/null +++ b/src/pci/host.rs @@ -0,0 +1,358 @@ +//! What a PCI host bridge supplies to its root bus, and what it does to a +//! CPU access on its way to a function. +//! +//! The bus (`pci/`) has no byte order. A host chooses how bytes move per +//! window as a [`ByteOrder`] value, and the physical bus maps [`BarDevice`] +//! entries that apply it, so the processor reaches a function with no +//! search of the bus and no host lock. The hosts (`mace_pci`, `bridge`) +//! implement [`PciHost`] and use the rest; the core files (`bus`, `decode`, +//! `intx`, `config`, `function`, `pci_to_pci_bridge`) use only the trait. + +use std::collections::HashMap; +use std::sync::{Arc, Mutex}; + +use crate::pci::{BarMapping, PciDma, PciFunction}; +use crate::traits::{BusDevice, BusRead16, BusRead32, BusRead64, BusRead8, BUS_OK}; + +/// What a root bus asks of its host bridge: the DMA each requester gets +/// and where each host IRQ goes. Nothing else. +pub trait PciHost: Send + Sync { + /// The DMA identity of root slot `requester`. + fn dma_for(&self, requester: u8) -> Arc; + /// Host IRQ `irq` is now `level`: the OR of every INTx line and board + /// IRQ routed to it. Calls are serialised and in order per IRQ. + fn set_irq(&self, irq: u8, level: bool); +} + +/// How a CPU window orders bytes on their way to a function. `Stream`: each +/// byte is at its own address whatever the access size. `Values`: each access +/// carries a whole value of `width` bytes, as the host's swap bit says. +/// +/// How a CPU access becomes bus cycles. Functions are little-endian and see +/// offsets and values as they would on the bus; the CPU is big-endian. +/// SGI's names for the two modes are byte stream and word values. +/// +/// `n` is the bytes in one cycle; the CPU's value is the number it loaded +/// or stores, so a little-endian function reading it back sees the same +/// number except where the table says the bytes are reversed. +/// +/// | Variant | Size | Address | Cycles | Placement | +/// |---|---|---|---|---| +/// | `Stream` | 1, 2, 4 | as given | one | PCI byte i is the value's i-th byte in memory order | +/// | `Stream` | 8 | as given | two, lower address first | the same, a word each | +/// | `Values{4}` | 1, 2 | `off ^ (4 - size)` | one | the value | +/// | `Values{4}` | 4 | as given | one | the value | +/// | `Values{4}` | 8 | as given | two, lower address first | the high word, then the low | +/// | `Values{8}` | 1, 2, 4 | `off ^ (8 - size)` | one | the value | +/// | `Values{8}` | 8 | as given | one (two word calls) | the low word, then the high | +/// +/// MACE is `Values{4}`: bytes `^3`, halves `^2`, words unchanged +/// (O2 Device Driver Programming Guide 007-3443-002, pp. 397-398, Table +/// 15-3, established). Its 64-bit PIO is not documented there; the two-word +/// split above is iris's existing behaviour and is unknown. The Bridge picks +/// per window from Device(x): its programmed swap bit gives `Stream`, +/// and its 64-bit device bit gives the width. +#[derive(Clone, Copy, PartialEq, Eq, Hash, Debug)] +pub enum ByteOrder { + Stream, + Values { width: u8 }, +} + +impl ByteOrder { + /// Word values for a bus `width` bytes wide (4 or 8). Anything else is a + /// machine bug. + pub fn values(width: u8) -> Self { + assert!(matches!(width, 4 | 8), "pci: no word-value byte order for a {width}-byte bus"); + ByteOrder::Values { width } + } + + /// A CPU read of `size` bytes (1, 2, 4 or 8) whose first byte is at BAR + /// offset `off`. + pub fn read(self, f: &dyn PciFunction, bar: usize, off: u32, size: usize) -> u64 { + match self { + ByteOrder::Stream => match size { + 1 => f.bar_read(bar, off, 1) as u64, + 2 => (f.bar_read(bar, off, 2) as u16).swap_bytes() as u64, + 4 => f.bar_read(bar, off, 4).swap_bytes() as u64, + _ => { + let first = f.bar_read(bar, off, 4).swap_bytes() as u64; + first << 32 | f.bar_read(bar, off.wrapping_add(4), 4).swap_bytes() as u64 + } + }, + ByteOrder::Values { width } => { + let width = width as usize; + if size == 8 { + let (a, b) = (f.bar_read(bar, off, 4) as u64, f.bar_read(bar, off.wrapping_add(4), 4) as u64); + return if width == 8 { b << 32 | a } else { a << 32 | b }; + } + f.bar_read(bar, off ^ (width - size) as u32, size) as u64 + } + } + } + + pub fn write(self, f: &dyn PciFunction, bar: usize, off: u32, size: usize, val: u64) { + match self { + ByteOrder::Stream => match size { + 1 => f.bar_write(bar, off, val as u32, 1), + 2 => f.bar_write(bar, off, (val as u16).swap_bytes() as u32, 2), + 4 => f.bar_write(bar, off, (val as u32).swap_bytes(), 4), + _ => { + f.bar_write(bar, off, ((val >> 32) as u32).swap_bytes(), 4); + f.bar_write(bar, off.wrapping_add(4), (val as u32).swap_bytes(), 4); + } + }, + ByteOrder::Values { width } => { + let width = width as usize; + if size == 8 { + let (hi, lo) = ((val >> 32) as u32, val as u32); + let (a, b) = if width == 8 { (lo, hi) } else { (hi, lo) }; + f.bar_write(bar, off, a, 4); + f.bar_write(bar, off.wrapping_add(4), b, 4); + return; + } + f.bar_write(bar, off ^ (width - size) as u32, val as u32, size); + } + } + } +} + +/// A run of a host's CPU window that one BAR decodes whole, as a bus device +/// the physical bus maps over the host's own window. `order` is by value: +/// the access is one `match` and the function's own call. +pub struct BarDevice { + f: Arc, + bar: usize, + /// The CPU address bits the host decodes, and where in them the BAR's + /// byte 0 sits (it may lie below the window: a BAR larger than it). + mask: u64, + base: u64, + order: ByteOrder, +} + +impl BarDevice { + pub fn new(f: Arc, bar: usize, base: u64, mask: u64, order: ByteOrder) -> Self { + Self { f, bar, mask, base, order } + } + + #[inline] + fn off(&self, addr: u64) -> u32 { + (addr & self.mask).wrapping_sub(self.base) as u32 + } +} + +impl BusDevice for BarDevice { + fn read8(&self, a: u64) -> BusRead8 { BusRead8::ok(self.order.read(&*self.f, self.bar, self.off(a), 1) as u8) } + fn read16(&self, a: u64) -> BusRead16 { BusRead16::ok(self.order.read(&*self.f, self.bar, self.off(a), 2) as u16) } + fn read32(&self, a: u64) -> BusRead32 { BusRead32::ok(self.order.read(&*self.f, self.bar, self.off(a), 4) as u32) } + fn read64(&self, a: u64) -> BusRead64 { BusRead64::ok(self.order.read(&*self.f, self.bar, self.off(a), 8)) } + fn write8(&self, a: u64, v: u8) -> u32 { self.order.write(&*self.f, self.bar, self.off(a), 1, v as u64); BUS_OK } + fn write16(&self, a: u64, v: u16) -> u32 { self.order.write(&*self.f, self.bar, self.off(a), 2, v as u64); BUS_OK } + fn write32(&self, a: u64, v: u32) -> u32 { self.order.write(&*self.f, self.bar, self.off(a), 4, v as u64); BUS_OK } + fn write64(&self, a: u64, v: u64) -> u32 { self.order.write(&*self.f, self.bar, self.off(a), 8, v); BUS_OK } +} + +/// The [`BarDevice`]s a host has made, one per distinct configuration. +/// +/// The physical bus keeps every device it is given for as long as it lives (its +/// decode table holds raw pointers), so a host that made a fresh `BarDevice` on +/// each publish would grow that list without bound, and every configuration +/// write updates the mappings. The same function, BAR, CPU base and order give +/// back the same `Arc`, so a configuration that is mapped again maps the entry +/// it already has. One cache per host: the CPU address mask is the host's and +/// is not part of the key. +#[derive(Default)] +pub struct BarDevices(Mutex>>); + +impl BarDevices { + /// The device that answers `run`, a run of a window with byte `order` + /// whose CPU address bits are `mask`. + pub fn get(&self, run: &BarMapping, mask: u64, order: ByteOrder) -> Arc { + let base = run.cpu.wrapping_sub(run.off as u64); + let key = (Arc::as_ptr(&run.function) as *const () as usize, run.bar, base, order); + self.0.lock().unwrap().entry(key).or_insert_with(|| Arc::new(BarDevice::new(run.function.clone(), run.bar, base, mask, order))).clone() + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::pci::testing::TestCard; + use crate::pci::Bar; + + const V: u64 = 0x0102_0304_0506_0708; + + fn card() -> Arc { + TestCard::new([Bar::Mem(0x1000), Bar::Unused, Bar::Unused, Bar::Unused, Bar::Unused, Bar::Unused], 0) + } + + /// The register file after a write of V's top `size` bytes at `off`, and + /// the BAR calls it took. + fn written(order: ByteOrder, off: u32, size: usize) -> (Vec, Vec<(u32, usize)>) { + let c = card(); + order.write(&*c, 0, off, size, V >> (64 - 8 * size)); + let bytes = c.mem()[..16].to_vec(); + (bytes, c.calls()) + } + + fn at(bytes: &[u8], off: usize, n: usize) -> Vec { + bytes[off..off + n].to_vec() + } + + #[test] + fn every_row_of_the_order_table() { + let be = |n: usize| V.to_be_bytes()[..n].to_vec(); + let le = |n: usize| (V >> (64 - 8 * n)).to_le_bytes()[..n].to_vec(); + // The value stored is V's top `size` bytes: 0x01, 0x0102, ... + for off in 0..8u32 { + let o = off as usize; + // Stream: address as given, bytes in memory order. + for size in [1usize, 2, 4, 8] { + if off as usize % size != 0 { continue } + let (b, calls) = written(ByteOrder::Stream, off, size); + assert_eq!(at(&b, o, size), be(size), "byte stream size {size} off {off}"); + let want: Vec<_> = if size == 8 { vec![(off, 4), (off + 4, 4)] } else { vec![(off, size)] }; + assert_eq!(calls, want, "cycles, byte stream size {size}"); + } + // Values{4}: sub-word accesses move to off ^ (4 - size), values unchanged. + for size in [1usize, 2, 4] { + if off as usize % size != 0 { continue } + let (b, calls) = written(ByteOrder::values(4), off, size); + let a = (off ^ (4 - size) as u32) as usize; + assert_eq!(at(&b, a, size), le(size), "word values 4 size {size} off {off}"); + assert_eq!(calls, vec![(a as u32, size)]); + } + if off % 8 == 0 { + let (b, calls) = written(ByteOrder::values(4), off, 8); + assert_eq!(at(&b, o, 4), ((V >> 32) as u32).to_le_bytes(), "high word first"); + assert_eq!(at(&b, o + 4, 4), (V as u32).to_le_bytes()); + assert_eq!(calls, vec![(off, 4), (off + 4, 4)]); + } + // Values{8}. + for size in [1usize, 2, 4] { + if off as usize % size != 0 { continue } + let (b, calls) = written(ByteOrder::values(8), off, size); + let a = (off ^ (8 - size) as u32) as usize; + assert_eq!(at(&b, a, size), le(size), "word values 8 size {size} off {off}"); + assert_eq!(calls, vec![(a as u32, size)]); + } + if off == 0 { + let (b, calls) = written(ByteOrder::values(8), 0, 8); + assert_eq!(at(&b, 0, 8), V.to_le_bytes(), "one cycle of eight reversed bytes"); + assert_eq!(calls, vec![(0, 4), (4, 4)], "carried as two word calls, low first"); + } + } + } + + /// Whatever a write leaves, the same access reads back. + #[test] + fn a_read_returns_what_the_same_write_stored() { + for order in [ByteOrder::Stream, ByteOrder::values(4), ByteOrder::values(8)] { + for size in [1usize, 2, 4, 8] { + for off in (0..16u32).step_by(size) { + let c = card(); + let v = if size == 8 { V } else { V >> (64 - 8 * size) }; + order.write(&*c, 0, off, size, v); + assert_eq!(order.read(&*c, 0, off, size), v, "{order:?} size {size} off {off}"); + } + } + } + } + + /// MACE's byte rule as it was written before ByteOrder: bytes `^3`, halves + /// `^2`, words as given. + #[test] + fn values_4_matches_mace_swizzle() { + let swizzle = |off: u32, width: usize| match width { + 1 => off ^ 3, + 2 => off ^ 2, + _ => off, + }; + for size in [1usize, 2, 4] { + for off in 0..32u32 { + let c = card(); + ByteOrder::values(4).read(&*c, 0, off, size); + let want = swizzle(off, size); + assert_eq!(c.calls(), vec![(want, size)], "size {size} off {off}"); + } + } + } + + /// The Bridge's old bar_cycle for a CPU access at `off`, copied here as + /// the reference: bytes in CPU order, swap programmed or not. + fn old_bridge_write(width: usize, swap: bool, off: u32, size: usize, val: u64, f: &dyn PciFunction) { + let mut host = [0u8; 8]; + for (k, b) in host.iter_mut().enumerate().take(size) { + *b = (val >> (8 * (size - 1 - k))) as u8; + } + let cycles: Vec<(u32, usize)> = if size > width { vec![(off, 4), (off + 4, 4)] } else { vec![(off, size)] }; + for (o, n) in cycles { + let start = if o == off { 0 } else { 4 }; + let h = &host[start..start + n]; + let pci: Vec = (0..n).map(|i| h[if swap { i } else { n - 1 - i }]).collect(); + let boff = if swap { o } else { o ^ (width - n) as u32 }; + let mut done = 0; + while done < n { + let chunk = (n - done).min(4); + let v = pci[done..done + chunk].iter().rev().fold(0u32, |a, &b| a << 8 | b as u32); + f.bar_write(0, boff + done as u32, v, chunk); + done += chunk; + } + } + } + + #[test] + fn values_matches_the_bridge_bar_cycle() { + for (order, width, swap) in [(ByteOrder::Stream, 4, true), (ByteOrder::Stream, 8, true), (ByteOrder::values(4), 4, false), (ByteOrder::values(8), 8, false)] { + for size in [1usize, 2, 4, 8] { + for off in (0..16u32).step_by(size) { + let (old, new) = (card(), card()); + let v = if size == 8 { V } else { V >> (64 - 8 * size) }; + old_bridge_write(width, swap, off, size, v, &*old); + order.write(&*new, 0, off, size, v); + assert_eq!(old.mem(), new.mem(), "{order:?} width {width} size {size} off {off}"); + } + } + } + } + + #[test] + #[should_panic(expected = "no word-value byte order")] + fn values_3_panics() { + ByteOrder::values(3); + } + + /// MACE's 64-bit PIO is not documented (the guide's table stops at 32 + /// bits); the two-word split, high word first, is iris's own choice and + /// this test pins it so a change is deliberate. + #[test] + fn mace_64_bit_is_the_two_word_split_and_marked_unknown() { + let c = card(); + ByteOrder::values(4).write(&*c, 0, 0x10, 8, 0x1122_3344_5566_7788); + assert_eq!(c.mem()[0x10..0x18], [0x44, 0x33, 0x22, 0x11, 0x88, 0x77, 0x66, 0x55]); + } + + #[test] + fn a_bar_device_applies_its_order_at_the_offset_from_its_base() { + let c = card(); + let bar = BarDevice::new(c.clone(), 0, 0x1a00_1000, 0xffff_ffff, ByteOrder::values(4)); + bar.write8(0x1a00_1084, 0xaa); + assert_eq!(c.calls(), vec![(0x87, 1)], "off 0x84, byte ^3"); + assert_eq!(bar.read32(0x1a00_1084).data, 0xaa00_0000, "the byte sits in the top byte of its word"); + let hi = BarDevice::new(c, 0, 0x1000, 0x01ff_ffff, ByteOrder::Stream); + hi.write16(0x9a00_1010, 0x1234); + assert_eq!(hi.read16(0x1a00_1010).data, 0x1234, "the host mask folds the aliases"); + } + + #[test] + fn a_configuration_that_is_made_again_is_the_same_device() { + let (c, cache) = (card(), BarDevices::default()); + let run = |cpu: u64, off: u32| BarMapping { cpu, len: 0x1000, function: c.clone(), bar: 0, off, window: 0 }; + let a = cache.get(&run(0x1a00_1000, 0), 0xffff_ffff, ByteOrder::values(4)); + // The same BAR placed at the same base, whichever page the run starts at. + let b = cache.get(&run(0x1a00_2000, 0x1000), 0xffff_ffff, ByteOrder::values(4)); + assert!(Arc::ptr_eq(&a, &b)); + assert!(!Arc::ptr_eq(&a, &cache.get(&run(0x1a00_1000, 0), 0xffff_ffff, ByteOrder::Stream)), "other order"); + assert!(!Arc::ptr_eq(&a, &cache.get(&run(0x1a01_1000, 0), 0xffff_ffff, ByteOrder::values(4))), "moved"); + assert!(!Arc::ptr_eq(&a, &cache.get(&run(0x1a00_1000, 0), 0xffff_ffff, ByteOrder::values(8))), "other width"); + } +} diff --git a/src/pci/intx.rs b/src/pci/intx.rs new file mode 100644 index 000000000..32e98198b --- /dev/null +++ b/src/pci/intx.rs @@ -0,0 +1,645 @@ +//! Interrupt pins and board IRQs, combined per host output. +//! +//! Every pin of every device number is a line (`device * 4 + pin - 1`) and +//! the board IRQs follow them. The combiner keeps each line's level and +//! tells the host the OR of everything routed to an output, so on a shared +//! output one source cannot clear another's interrupt. +//! +//! No lock of ours is held while the host is driven. A level change is one +//! `fetch_or` or `fetch_and` on the line's word and one `fetch_or` on the +//! output's `state` word. `state` carries everything the delivery handshake +//! needs: `DIRTY` (a level changed), `HELD` (someone is calling the host), +//! `FELL` and `FORCE`. A mutator sets `DIRTY | HELD` in one RMW and returns if +//! `HELD` was already set. The holder takes the flags with a `fetch_and`, +//! reads the OR, calls the host, and gives up with a `compare_exchange` of +//! `HELD` to 0 that fails if any flag was set meanwhile, in which case it +//! goes round again. So deliveries to one output are serialised and in order, +//! the last one is the OR after the last change, and every step of the +//! handshake is a read-modify-write on one location: acquire-release is +//! enough, with no store-then-load pair for a store buffer to reorder. That +//! is checked by loom (`loom_tests`, built with `RUSTFLAGS="--cfg loom"`), +//! since x86 hardware cannot show a missing acquire or release. +//! +//! A fall followed by a rise before the deliverer reads the OR would reach the +//! host as a steady 1. That is harmless to a level host and wrong for an edge +//! host (the Bridge sends HEART a packet only on a rising edge), so a change +//! that takes an output from 1 to 0 sets `FELL` with its `DIRTY` and the +//! deliverer sends 0 then 1. A rise followed by a fall that nobody saw is +//! coalesced. + +use std::sync::{Arc, Weak}; + +use crate::irq::{InterruptController, IrqLine}; +use crate::pci::PciHost; + +#[cfg(loom)] +use loom::sync::atomic::{AtomicBool, AtomicU64, AtomicU8, Ordering::*}; +#[cfg(not(loom))] +use std::sync::atomic::{AtomicBool, AtomicU64, AtomicU8, Ordering::*}; + +/// A board interrupt that is not a PCI pin but shares a host input, for +/// example one chip's interrupt on a Bridge pin. +#[derive(Clone, Copy, PartialEq, Eq, Debug)] +pub struct BoardIrq(pub u8); + +/// One root bus's interrupt routing as plain data. Labels and sources go +/// in comments beside the static. +pub struct IntxRouting { + /// Device numbers on the bus (at most 32). + pub devices: u8, + /// Host IRQs (CRIME inputs on the O2, Bridge pins on the Octane). + pub irqs: u8, + /// Device to host IRQ for INTA..INTD; a pin not listed is not routed. + pub intx: &'static [(u8, [Option; 4])], + /// Host IRQ of `BoardIrq(i)`. + pub board_irqs: &'static [u8], +} + +/// Lines `0..PIN_LINES` are pins; board IRQ `n` is line `PIN_LINES + n`. Line +/// `l` is bit `l % 64` of word `l / 64`, so the board IRQs are the third word. +const PIN_LINES: u32 = 32 * 4; + +/// Bits of [`Out::state`]. +const DIRTY: u8 = 1; // a level changed since the holder last took the flags +const HELD: u8 = 2; // someone is calling the host for this output +const FELL: u8 = 4; // the output went 1 to 0 since the last delivery +const FORCE: u8 = 8; // send the current level even though the host has it + +/// What drives one output, and its delivery state. +struct Out { + /// The lines routed here, by word. + mask: [u64; 3], + state: AtomicU8, + /// The level the host has; the holder's alone. + last: AtomicBool, +} + +/// Where a test hook runs: after a lower's `fetch_and` and before it works +/// out the output's level, or just before the holder gives up `HELD`. +#[derive(Clone, Copy, PartialEq, Eq, Debug)] +enum Point { + Lowered, + Releasing, +} + +pub(super) struct Combiner { + up: Weak, + /// Every line's level: pins in words 0 and 1, board IRQs in word 2. + lines: [AtomicU64; 3], + /// Per device, the output of INTA..INTD. + route: Box<[[Option; 4]]>, + board_irq_out: &'static [u8], + out: Box<[Out]>, + /// Lets a test act at the two points a concurrent change can land. + #[cfg(test)] + hook: std::sync::Mutex>>, +} + +impl Combiner { + pub(super) fn new(w: &'static IntxRouting, up: Weak) -> Arc { + assert!(w.devices <= 32 && w.board_irqs.len() <= 32, "pci: routing has too many devices or board IRQs"); + let mut route = vec![[None; 4]; w.devices as usize]; + let mut out: Vec = + (0..w.irqs).map(|_| Out { mask: [0; 3], state: AtomicU8::new(0), last: AtomicBool::new(false) }).collect(); + let target = |o: u8| { + assert!(o < w.irqs, "pci: routing names output {o} of {}", w.irqs); + o as usize + }; + for &(dev, outs) in w.intx { + assert!(dev < w.devices, "pci: wiring names device {dev} of {}", w.devices); + route[dev as usize] = outs; + for (p, o) in outs.iter().enumerate() { + if let Some(o) = *o { + let line = dev as usize * 4 + p; + out[target(o)].mask[line / 64] |= 1 << (line % 64); + } + } + } + for (i, &o) in w.board_irqs.iter().enumerate() { + out[target(o)].mask[2] |= 1 << i; + } + Arc::new(Self { + up, + lines: Default::default(), + route: route.into(), + board_irq_out: w.board_irqs, + out: out.into(), + #[cfg(test)] + hook: Default::default(), + }) + } + + /// The line of device `dev`'s pin (1 = INTA), or None if the wiring + /// does not route it. + pub(super) fn pin_line(self: &Arc, dev: u8, pin: u8) -> Option { + let routed = self.route.get(dev as usize)?.get(pin as usize - 1)?.is_some(); + routed.then(|| IrqLine::new(self.clone(), dev as u32 * 4 + pin as u32 - 1)) + } + + /// The output device `dev`'s pin (1 = INTA) is routed to. + pub(super) fn output(&self, dev: u8, pin: u8) -> Option { + *self.route.get(dev as usize)?.get(pin as usize - 1)? + } + + pub(super) fn board_irq_line(self: &Arc, w: BoardIrq) -> IrqLine { + assert!((w.0 as usize) < self.board_irq_out.len(), "pci: no board IRQ {}", w.0); + IrqLine::new(self.clone(), PIN_LINES + w.0 as u32) + } + + /// The output line `line` is routed to, if any. + fn output_of(&self, line: u32) -> Option { + let o = if line < PIN_LINES { + (*self.route.get((line / 4) as usize)?.get((line % 4) as usize)?)? + } else { + *self.board_irq_out.get((line - PIN_LINES) as usize)? + }; + Some(o as usize) + } + + /// Whether anything routed to an output is high, given word `i` as it + /// is now unless `known` supplies it. + fn any_high(&self, m: &Out, known: Option<(usize, u64)>) -> bool { + (0..3).any(|i| { + let word = match known { + Some((k, w)) if k == i => w, + _ => self.lines[i].load(Relaxed), + }; + word & m.mask[i] != 0 + }) + } + + fn set_line_level(&self, line: u32, level: bool) { + let Some(o) = self.output_of(line) else { return }; + let (w, bit) = ((line / 64) as usize, 1u64 << (line % 64)); + if level { + self.lines[w].fetch_or(bit, Relaxed); + return self.kick(o, 0); + } + let old = self.lines[w].fetch_and(!bit, Relaxed); + self.at(Point::Lowered); + // Whether the output fell, from `old` and not from a later read of + // this word, so a rise that follows cannot hide the fall. (A rise in + // another word is read as it is now; that is the limit of doing this + // without a lock.) + let fell = old & bit != 0 && !self.any_high(&self.out[o], Some((w, old & !bit))); + self.kick(o, if fell { FELL } else { 0 }); + } + + #[cfg(test)] + fn at(&self, p: Point) { + // Not under the lock: a hook may change levels, which comes back here. + let h = self.hook.lock().unwrap().clone(); + if let Some(h) = h { + h(p); + } + } + + #[cfg(not(test))] + #[inline(always)] + fn at(&self, _: Point) {} + + /// Deliver output `o` if nobody else is; whoever is will see `DIRTY`. + fn kick(&self, o: usize, flags: u8) { + let m = &self.out[o]; + if m.state.fetch_or(DIRTY | HELD | flags, AcqRel) & HELD != 0 { + return; + } + loop { + let s = m.state.fetch_and(!(DIRTY | FELL | FORCE), AcqRel); + let v = self.any_high(m, None); + let mut last = m.last.load(Relaxed); + if let Some(up) = self.up.upgrade() { + if s & FELL != 0 && v && last { + up.set_irq(o as u8, false); + last = false; + } + if s & FORCE != 0 || v != last { + up.set_irq(o as u8, v); + } + } + m.last.store(v, Relaxed); + self.at(Point::Releasing); + // Fails if a change landed meanwhile: go again. + if m.state.compare_exchange(HELD, 0, AcqRel, Relaxed).is_ok() { + return; + } + } + } + + /// Send every output's current level to the host, whether or not it + /// already has it. + pub(super) fn redrive(&self) { + for o in 0..self.out.len() { + self.kick(o, FORCE); + } + } + + /// Each output's (dirty, fell, last), for a test to compare. + #[cfg(test)] + pub(super) fn flags(&self) -> Vec<(bool, bool, bool)> { + self.out.iter().map(|m| { + let s = m.state.load(SeqCst); + (s & DIRTY != 0, s & FELL != 0, m.last.load(SeqCst)) + }).collect() + } + + /// Latch every output's `FELL` and `DIRTY`, as a delivery interrupted + /// mid-flight would leave them. + #[cfg(test)] + pub(super) fn out_force_flags(&self) { + for m in self.out.iter() { + m.state.fetch_or(FELL | DIRTY, SeqCst); + } + } + + /// The pin words and the wire word, for a snapshot. + pub(super) fn save(&self) -> (u64, u64, u32) { + (self.lines[0].load(SeqCst), self.lines[1].load(SeqCst), self.lines[2].load(SeqCst) as u32) + } + + /// Replace the levels outright and forget delivery state: what the host + /// has is whatever the restore gave it, and `redrive` makes it agree. + pub(super) fn load(&self, w: (u64, u64, u32)) { + self.lines[0].store(w.0, SeqCst); + self.lines[1].store(w.1, SeqCst); + self.lines[2].store(w.2 as u64, SeqCst); + for m in self.out.iter() { + m.state.store(0, SeqCst); + m.last.store(false, Relaxed); + } + } +} + +impl InterruptController for Combiner { + fn set_line(&self, line: u32, level: bool) { + self.set_line_level(line, level); + } +} + +#[cfg(all(test, not(loom)))] +mod tests { + use super::*; + use crate::pci::PciDma; + use std::sync::mpsc::{channel, Receiver, Sender}; + use std::sync::Mutex; + + struct Nothing; + impl PciDma for Nothing { + fn dma_read(&self, _: u64, _: &mut [u8]) -> crate::pci::PciDmaStatus { crate::pci::PciDmaStatus::Ok } + fn dma_write(&self, _: u64, _: &[u8]) -> crate::pci::PciDmaStatus { crate::pci::PciDmaStatus::Ok } + } + + /// Records each delivery. With `stall` armed the next delivery reports + /// that it has started and then waits to be released, so a test can + /// change levels while a delivery is in flight. + struct Host { + log: Mutex>, + in_flight: Mutex, + overlapped: AtomicBool, + stall: Mutex, Receiver<()>)>>, + } + + impl Host { + fn new() -> Arc { + Arc::new(Self { + log: Mutex::new(Vec::new()), + in_flight: Mutex::new(0), + overlapped: AtomicBool::new(false), + stall: Mutex::new(None), + }) + } + + /// Arm one stall; returns the signal that it has begun and the release. + fn arm(&self) -> (Receiver<()>, Sender<()>) { + let (begun_tx, begun_rx) = channel(); + let (go_tx, go_rx) = channel(); + *self.stall.lock().unwrap() = Some((begun_tx, go_rx)); + (begun_rx, go_tx) + } + + fn log(&self) -> Vec<(u8, bool)> { + self.log.lock().unwrap().clone() + } + } + + impl PciHost for Host { + fn dma_for(&self, _: u8) -> Arc { + Arc::new(Nothing) + } + fn set_irq(&self, output: u8, level: bool) { + { + let mut n = self.in_flight.lock().unwrap(); + *n += 1; + if *n > 1 { + self.overlapped.store(true, SeqCst); + } + } + self.log.lock().unwrap().push((output, level)); + let stall = self.stall.lock().unwrap().take(); + if let Some((begun, go)) = stall { + begun.send(()).unwrap(); + go.recv().unwrap(); + } + *self.in_flight.lock().unwrap() -= 1; + } + } + + // Device 1 INTA and device 2 INTA share output 0; device 1 INTB is + // output 1; device 3 has INTA..INTD on 2..5; wire 0 is output 0 and + // wire 1 output 6 and wire 2 output 0. + static W: IntxRouting = IntxRouting { + devices: 8, + irqs: 7, + intx: &[ + (1, [Some(0), Some(1), None, None]), + (2, [Some(0), None, None, None]), + (3, [Some(2), Some(3), Some(4), Some(5)]), + ], + board_irqs: &[0, 6, 0], + }; + + fn rig() -> (Arc, Arc) { + let h = Host::new(); + let up: Weak = Arc::downgrade(&(h.clone() as Arc)); + (h, Combiner::new(&W, up)) + } + + #[test] + fn every_device_pin_reaches_its_output() { + let (h, c) = rig(); + for (dev, pin, out) in [(1, 1, 0), (1, 2, 1), (2, 1, 0), (3, 1, 2), (3, 2, 3), (3, 3, 4), (3, 4, 5)] { + let l = c.pin_line(dev, pin).unwrap(); + l.set(true); + assert_eq!(h.log().last(), Some(&(out, true)), "device {dev} pin {pin}"); + l.set(false); + assert_eq!(h.log().last(), Some(&(out, false)), "device {dev} pin {pin}"); + } + } + + #[test] + fn unrouted_pin_line_is_none() { + let (_, c) = rig(); + assert!(c.pin_line(1, 3).is_none(), "INTC of device 1"); + assert!(c.pin_line(4, 1).is_none(), "device 4 has nothing"); + assert!(c.pin_line(9, 1).is_none(), "past the last device"); + } + + #[test] + fn shared_pins_or_and_edges_only_are_delivered() { + let (h, c) = rig(); + let (a, b) = (c.pin_line(1, 1).unwrap(), c.pin_line(2, 1).unwrap()); + a.set(true); + b.set(true); + a.set(false); + assert_eq!(h.log(), vec![(0, true)], "b still holds it, and a second rise is not an edge"); + b.set(false); + assert_eq!(h.log(), vec![(0, true), (0, false)]); + b.set(false); + assert_eq!(h.log().len(), 2, "lowering a low pin delivers nothing"); + } + + #[test] + fn a_board_irq_reaches_its_output_and_ors_with_pins() { + let (h, c) = rig(); + let w = c.board_irq_line(BoardIrq(0)); + let p = c.pin_line(1, 1).unwrap(); + w.set(true); + p.set(true); + w.set(false); + assert_eq!(h.log(), vec![(0, true)], "the pin still holds output 0"); + p.set(false); + assert_eq!(h.log(), vec![(0, true), (0, false)]); + c.board_irq_line(BoardIrq(1)).set(true); + assert_eq!(h.log().last(), Some(&(6, true))); + } + + /// Run `op` on a thread whose delivery stalls inside the host, change + /// levels from the test thread meanwhile, then let the delivery go. + fn stalled(h: &Arc, op: impl FnOnce() + Send + 'static, during: impl FnOnce()) { + let (begun, go) = h.arm(); + let t = std::thread::spawn(op); + begun.recv().unwrap(); + during(); + go.send(()).unwrap(); + t.join().unwrap(); + } + + #[test] + fn a_pin_fall_then_rise_before_delivery_gives_zero_then_one() { + let (h, c) = rig(); + let p = c.pin_line(1, 1).unwrap(); + let raise = p.clone(); + stalled(&h, move || raise.set(true), || { + p.set(false); + p.set(true); + }); + assert_eq!(h.log(), vec![(0, true), (0, false), (0, true)]); + } + + #[test] + fn a_shared_output_with_the_other_pin_high_gets_no_zero_then_one() { + let (h, c) = rig(); + let (p, other) = (c.pin_line(1, 1).unwrap(), c.pin_line(2, 1).unwrap()); + let raise = p.clone(); + stalled(&h, move || raise.set(true), || { + other.set(true); + p.set(false); + p.set(true); + }); + assert_eq!(h.log(), vec![(0, true)], "the output never fell"); + } + + #[test] + fn a_raise_lowered_before_delivery_is_coalesced() { + let (h, c) = rig(); + let (p, q) = (c.pin_line(1, 1).unwrap(), c.pin_line(2, 1).unwrap()); + p.set(true); + let lower = p.clone(); + stalled(&h, move || lower.set(false), || { + q.set(true); + q.set(false); + }); + assert_eq!(h.log(), vec![(0, true), (0, false)], "the host never sees q's pulse"); + } + + #[test] + fn deliveries_are_ordered_and_never_concurrent() { + let (h, c) = rig(); + let pins: Vec<_> = [(1, 1), (2, 1)].iter().map(|&(d, p)| c.pin_line(d, p).unwrap()).collect(); + let ts: Vec<_> = pins + .into_iter() + .map(|l| { + std::thread::spawn(move || { + for _ in 0..2000 { + l.set(true); + l.set(false); + } + }) + }) + .collect(); + for t in ts { + t.join().unwrap(); + } + assert!(!h.overlapped.load(SeqCst)); + let log = h.log(); + assert_eq!(log.last().map(|e| e.1), Some(false), "ends low"); + assert!(log.windows(2).all(|w| w[0].1 != w[1].1), "levels alternate"); + } + + #[test] + fn redrive_resends_a_high_output_equal_to_last() { + let (h, c) = rig(); + c.pin_line(1, 1).unwrap().set(true); + c.redrive(); + assert_eq!(&h.log()[..2], &[(0, true), (0, true)]); + assert_eq!(h.log().iter().filter(|e| e.0 == 0).count(), 2); + assert_eq!(h.log().len(), 2 + 6, "and every other output, low"); + } + + #[test] + fn a_board_irq_fall_keeps_fell() { + let (h, c) = rig(); + let w = c.board_irq_line(BoardIrq(0)); + let raise = w.clone(); + stalled(&h, move || raise.set(true), || { + w.set(false); + w.set(true); + }); + assert_eq!(h.log(), vec![(0, true), (0, false), (0, true)]); + } + + #[test] + fn load_resets_dirty_fell_and_last() { + let (h, c) = rig(); + let p = c.pin_line(1, 1).unwrap(); + p.set(true); + let saved = c.save(); + // A fall that nobody delivered: `fell` is latched and `dirty` set. + c.out[0].state.fetch_or(FELL | DIRTY, SeqCst); + p.set(false); + c.load(saved); + assert_eq!(c.save(), saved); + assert_eq!(c.out[0].state.load(SeqCst), 0, "no flag left"); + assert!(!c.out[0].last.load(SeqCst)); + // After the load the host is told the saved level whatever it held. + let n = h.log().len(); + c.redrive(); + assert_eq!(h.log()[n], (0, true)); + } + + /// A mutator that has set `DIRTY` and found `HELD` set returns; the + /// holder must see the flag when it tries to give up. The hook plays that + /// mutator in the window just before the holder does. + #[test] + fn a_hook_lower_just_before_the_holder_gives_up_is_still_delivered() { + let (h, c) = rig(); + let p = c.pin_line(1, 1).unwrap(); + let fired = Arc::new(AtomicBool::new(false)); + { + let (fired, c2) = (fired.clone(), c.clone()); + *c.hook.lock().unwrap() = Some(Arc::new(move |at| { + if at == Point::Releasing && !fired.swap(true, SeqCst) { + c2.lines[0].fetch_and(!(1 << 4), SeqCst); + let held = c2.out[0].state.fetch_or(DIRTY | HELD, SeqCst) & HELD != 0; + assert!(held, "the holder still holds the output"); + } + })); + } + p.set(true); + assert!(fired.load(SeqCst)); + assert_eq!(h.log(), vec![(0, true), (0, false)]); + } + + /// Another source rises between a lower's `fetch_and` and its work on + /// the output's level. The lower took the output to 0 (and the rise + /// took it back), so the host must see 0 then 1; a level read after the + /// lower would find the output high and see no fall. (A rise in the + /// other word, or from the other kind of source, is read as it is when + /// the lower works out the level; that is the limit of doing this + /// without a lock.) + #[test] + fn a_rise_between_the_lower_and_its_level_still_latches_the_fall() { + for wire in [false, true] { + let (h, c) = rig(); + let (x, y) = if wire { + (c.board_irq_line(BoardIrq(0)), c.board_irq_line(BoardIrq(2))) + } else { + (c.pin_line(1, 1).unwrap(), c.pin_line(2, 1).unwrap()) + }; + x.set(true); + let fired = Arc::new(AtomicBool::new(false)); + let fired2 = fired.clone(); + *c.hook.lock().unwrap() = Some(Arc::new(move |at| { + if at == Point::Lowered && !fired2.swap(true, SeqCst) { + y.set(true); + } + })); + x.set(false); + assert_eq!(h.log(), vec![(0, true), (0, false), (0, true)], "wire: {wire}"); + } + } +} + +/// Every interleaving of a few threads changing lines, checked by loom (run +/// with `RUSTFLAGS="--cfg loom"`). The handshake is read-modify-writes on +/// one word, so acquire-release is what these runs check; the tests above +/// cannot, because x86 hardware does not reorder such operations. +#[cfg(all(test, loom))] +mod loom_tests { + use super::*; + use crate::pci::PciDma; + use loom::sync::atomic::{AtomicBool, AtomicU32}; + use loom::sync::Arc; + use loom::thread; + + /// Counts deliveries, remembers the last level, and notices two + /// deliveries to one output at once. + struct Host { + last: AtomicBool, + inside: AtomicU32, + overlapped: AtomicBool, + } + + impl PciHost for Host { + fn dma_for(&self, _: u8) -> std::sync::Arc { + unreachable!() + } + fn set_irq(&self, _: u8, level: bool) { + if self.inside.fetch_add(1, SeqCst) != 0 { + self.overlapped.store(true, SeqCst); + } + self.last.store(level, SeqCst); + self.inside.fetch_sub(1, SeqCst); + } + } + + static W: IntxRouting = IntxRouting { devices: 4, irqs: 1, intx: &[(1, [Some(0), None, None, None]), (2, [Some(0), None, None, None])], board_irqs: &[] }; + + /// Two sources of one output change at once. Whatever the order, the + /// last level the host hears is the output's level, and no two + /// deliveries overlap. + #[test] + fn the_host_ends_with_the_outputs_level() { + loom::model(|| { + let host = std::sync::Arc::new(Host { last: AtomicBool::new(false), inside: AtomicU32::new(0), overlapped: AtomicBool::new(false) }); + let up: Weak = std::sync::Arc::downgrade(&(host.clone() as std::sync::Arc)); + let c = Combiner::new(&W, up); + let (a, b) = (c.pin_line(1, 1).unwrap(), c.pin_line(2, 1).unwrap()); + a.set(true); + let t = { + let a = a.clone(); + thread::spawn(move || { + a.set(false); + a.set(true); + }) + }; + b.set(true); + b.set(false); + t.join().unwrap(); + assert!(!host.overlapped.load(SeqCst)); + let level = c.lines[0].load(SeqCst) & c.out[0].mask[0] != 0; + assert_eq!(host.last.load(SeqCst), level); + assert_eq!(c.out[0].state.load(SeqCst), 0, "nothing left pending"); + }); + } +} diff --git a/src/pci/pci_to_pci_bridge.rs b/src/pci/pci_to_pci_bridge.rs new file mode 100644 index 000000000..43fa0fe5f --- /dev/null +++ b/src/pci/pci_to_pci_bridge.rs @@ -0,0 +1,281 @@ +//! A PCI-to-PCI bridge: a function with a type 1 header and a bus behind it. +//! +//! Firmware numbers the bus (secondary and subordinate bus numbers) and +//! programs the I/O, memory and prefetchable windows; the bridge forwards the +//! BARs of the functions behind it that lie inside an enabled window, so the +//! bus in front decodes them like its own. A pin of the function at device +//! `d` behind the bridge reaches pin `(pin + d) mod 4` of the bridge's own +//! device, and its DMA is the bridge's. + +use std::sync::Arc; + +use super::{DevFn, PciBus, PciConfig, PciFunction, PciFunctionState}; + +/// Bridge control bit 6: secondary bus reset. +const SECONDARY_RESET: u32 = 1 << 22; + +pub struct PciToPciBridge { + pci: PciFunctionState, + sec: PciBus, +} + +impl PciToPciBridge { + /// A bridge with PCI identity `vendor` and `device`, to be fitted at `at` + /// on `parent`, and the bus behind it. The caller attaches it with + /// `parent.attach(at, bridge)`. Only the type 1 header is modelled, not + /// the registers of any chip, so the identity is the machine's to name. + pub fn new(parent: &PciBus, at: DevFn, vendor: u16, device: u16) -> Arc { + Arc::new(Self { + pci: PciFunctionState::new(PciConfig::type1(vendor, device, 0)), + sec: PciBus::secondary(parent.clone(), at), + }) + } +} + +impl PciToPciBridge { + /// The bus behind the bridge. + pub fn secondary(&self) -> &PciBus { + &self.sec + } +} + +impl PciFunction for PciToPciBridge { + fn state(&self) -> &PciFunctionState { + &self.pci + } + + fn bar_read(&self, _bar: usize, _off: u32, _width: usize) -> u32 { + 0 + } + + fn bar_write(&self, _bar: usize, _off: u32, _val: u32, _width: usize) {} + + + fn secondary(&self) -> Option<&PciBus> { + Some(&self.sec) + } + + /// A write that raises secondary bus reset resets the bus behind the + /// bridge and nothing else. + fn config_written(&self, reg: usize, old: u32, new: u32) { + if reg == 15 && old & SECONDARY_RESET == 0 && new & SECONDARY_RESET != 0 { + self.sec.reset(); + } + } + + /// RST# on the primary bus reaches everything behind the bridge. + fn reset(&self) { + self.sec.reset(); + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::pci::testing::{TestCard, TestHost}; + use crate::pci::{Bar, CpuWindow, IntxRouting, PciHost, Space, PCI_COMMAND_MEMORY}; + use std::sync::{Mutex, Weak}; + + // The bridge sits at device 3, whose INTA..INTD reach outputs 0 to 3. + static W: IntxRouting = IntxRouting { + devices: 8, + irqs: 4, + intx: &[(3, [Some(0), Some(1), Some(2), Some(3)]), (4, [Some(0), None, None, None])], + board_irqs: &[], + }; + + fn at(device: u8, function: u8) -> DevFn { + DevFn { device, function } + } + + fn bars() -> [Bar; 6] { + [Bar::Mem(0x1000), Bar::Unused, Bar::Unused, Bar::Unused, Bar::Unused, Bar::Unused] + } + + struct Rig { + host: Arc, + root: PciBus, + bridge: Arc, + } + + /// A bridge at device 3 numbering a bus 1 behind it, with window + /// registers programmed for memory 0x8000_0000..0x800f_ffff. + fn rig() -> Rig { + let host = Arc::new(TestHost::default()); + let up: Weak = Arc::downgrade(&(host.clone() as Arc)); + let root = PciBus::new(0, &W, up); + let bridge = PciToPciBridge::new(&root, at(3, 0), 0x1234, 0x5678); + root.attach(at(3, 0), bridge.clone()); + root.config_write(at(3, 0), 0x18, 4, 0x0001_0100); // primary 0, secondary 1, subordinate 1 + root.config_write(at(3, 0), 0x20, 4, 0x8000_8000); + root.config_write(at(3, 0), 0x04, 2, PCI_COMMAND_MEMORY); + Rig { host, root, bridge } + } + + /// A card on the secondary bus whose BAR 0 is at `base`, decoding on. + fn child(r: &Rig, device: u8, pin: u8, base: u32) -> Arc { + let c = TestCard::new(bars(), pin); + let sec = r.bridge.secondary(); + sec.attach(at(device, 0), c.clone()); + sec.config_write(at(device, 0), 0x10, 4, base); + sec.config_write(at(device, 0), 0x04, 2, PCI_COMMAND_MEMORY); + c + } + + fn windows() -> Vec { + let w = |space, cpu, pci, len| CpuWindow { space, cpu, pci, len, mask: 0xffff_ffff }; + vec![w(Space::Io, 0x1800_0000, 0, 0x200_0000), w(Space::Mem, 0x1a00_0000, 0x8000_0000, 0x200_0000)] + } + + fn cpus(r: &Rig) -> Arc>> { + let seen = Arc::new(Mutex::new(Vec::new())); + let sink = seen.clone(); + r.root.on_bar_mappings(Box::new(move |runs| *sink.lock().unwrap() = runs.iter().map(|b| (b.cpu, b.window)).collect())); + r.root.set_host_windows(&windows); + seen + } + + #[test] + fn type_1_header_probe_and_window_registers() { + let host = Arc::new(TestHost::default()); + let up: Weak = Arc::downgrade(&(host.clone() as Arc)); + let root = PciBus::new(0, &W, up); + root.attach(at(3, 0), PciToPciBridge::new(&root, at(3, 0), 0x1234, 0x5678)); + let read = |off, w| root.config_read(at(3, 0), off, w).unwrap(); + assert_eq!(read(0x0e, 1), 1, "header type 1"); + assert_eq!(read(0x09, 3), 0x06_0400, "a PCI-to-PCI bridge"); + for (off, probe, stuck) in [(0x18, 0xffff_ffffu32, 0xffff_ffffu32), (0x1c, !0, 0xf0f0), (0x20, !0, 0xfff0_fff0), (0x24, !0, 0xfff0_fff0), (0x28, !0, 0), (0x2c, !0, 0), (0x30, !0, 0), (0x38, !0, 0)] { + root.config_write(at(3, 0), off, 4, probe); + assert_eq!(read(off, 4), stuck, "register {off:#x}"); + } + root.config_write(at(3, 0), 0x10, 4, !0); + assert_eq!(read(0x10, 4), 0, "no BARs"); + root.config_write(at(3, 0), 0x19, 1, 0x05); + assert_eq!(read(0x19, 1), 0x05, "the secondary bus number is a byte"); + } + + #[test] + fn a_child_bar_inside_the_bridge_window_gets_a_run_with_the_parents_window_index() { + let r = rig(); + let seen = cpus(&r); + child(&r, 2, 0, 0x8000_2000); + assert_eq!(*seen.lock().unwrap(), vec![(0x1a00_2000, 1)]); + let (_, bar, off) = r.root.decode(Space::Mem, 0x8000_2010).expect("the slow path finds it too"); + assert_eq!((bar, off), (0, 0x10)); + } + + /// Firmware opens the bridge's memory window again, wider, over a child + /// BAR that lay outside the old window (a second enumeration with no + /// reset in between): the child gains its run and the slow path finds + /// it, because a window write updates the bus in front. + #[test] + fn widening_the_window_over_an_unmoved_bar_publishes_the_child() { + let r = rig(); + let seen = cpus(&r); + child(&r, 0, 0, 0x8080_0000); + assert_eq!(*seen.lock().unwrap(), vec![], "outside the 1 MB window"); + assert!(r.root.decode(Space::Mem, 0x8080_0008).is_none()); + r.root.config_write(at(3, 0), 0x22, 2, 0x80f0); + assert_eq!(*seen.lock().unwrap(), vec![(0x1a80_0000, 1)]); + assert_eq!(r.root.decode(Space::Mem, 0x8080_0008).map(|(_, b, o)| (b, o)), Some((0, 8))); + } + + #[test] + fn a_child_bar_outside_the_window_has_no_run_and_no_bar_access() { + let r = rig(); + let seen = cpus(&r); + let c = child(&r, 2, 0, 0x9000_0000); + assert!(seen.lock().unwrap().is_empty()); + assert!(r.root.decode(Space::Mem, 0x9000_0010).is_none()); + assert!(r.bridge.secondary().decode(Space::Mem, 0x9000_0010).is_some(), "the bus behind still decodes it"); + let _ = c; + } + + #[test] + fn disabling_the_bridge_memory_enable_removes_child_runs() { + let r = rig(); + let seen = cpus(&r); + child(&r, 2, 0, 0x8000_2000); + assert_eq!(seen.lock().unwrap().len(), 1); + r.root.config_write(at(3, 0), 0x04, 2, 0); + assert!(seen.lock().unwrap().is_empty()); + assert!(r.root.decode(Space::Mem, 0x8000_2010).is_none()); + r.root.config_write(at(3, 0), 0x04, 2, PCI_COMMAND_MEMORY); + assert_eq!(seen.lock().unwrap().len(), 1, "and enabling it brings the run back"); + } + + /// A pin of the function at device `d` behind the bridge reaches the + /// bridge's pin `(pin + d) mod 4`: device 2 INTA is the bridge's INTC. + #[test] + fn a_child_interrupt_reaches_parent_pin_p_plus_d_mod_4() { + for (device, pin, output) in [(2u8, 1u8, 2u32), (2, 2, 3), (2, 3, 0), (1, 1, 1), (0, 4, 3)] { + let r = rig(); + let c = child(&r, device, pin, 0x8000_1000); + c.state().irq().unwrap().set(true); + assert_eq!(r.host.irqs(), vec![(output as u8, true)], "device {device} pin {pin}"); + } + } + + #[test] + fn a_child_dma_uses_the_bridges_root_slot() { + let r = rig(); + let c = child(&r, 2, 0, 0x8000_2000); + let mut b = [0u8; 1]; + assert!(c.state().dma().unwrap().dma_read(0, &mut b).is_ok()); + assert_eq!(b[0], 3, "the bridge's device number, not the child's"); + } + + #[test] + fn route_finds_the_bus_by_secondary_and_subordinate() { + let r = rig(); + let inner = PciToPciBridge::new(r.bridge.secondary(), at(1, 0), 0x1234, 0x5678); + r.bridge.secondary().attach(at(1, 0), inner.clone()); + // Bus 1 holds bridge 1.0, which numbers bus 2; the outer one takes 1 to 2. + r.root.config_write(at(3, 0), 0x18, 4, 0x0002_0100); + r.bridge.secondary().config_write(at(1, 0), 0x18, 4, 0x0002_0201); + assert_eq!(r.root.find_bus(0).unwrap().number(), r.root.number()); + assert_eq!(r.root.find_bus(1).unwrap().number(), r.bridge.secondary().number()); + assert_eq!(r.root.find_bus(2).unwrap().number(), inner.secondary().number()); + assert!(r.root.find_bus(3).is_none()); + } + + /// Raising secondary bus reset resets what is behind the bridge and + /// nothing else. + #[test] + fn bridge_control_bit_6_resets_the_secondary_only() { + let r = rig(); + let log = Arc::new(Mutex::new(Vec::new())); + let behind = TestCard::logged("behind", bars(), 0, &log); + r.bridge.secondary().attach(at(2, 0), behind); + r.root.attach(at(5, 0), TestCard::logged("beside", bars(), 0, &log)); + r.root.config_write(at(3, 0), 0x3e, 2, 0x40); + assert_eq!(*log.lock().unwrap(), ["behind cmd=0x0"]); + r.root.config_write(at(3, 0), 0x3e, 2, 0x40); + assert_eq!(log.lock().unwrap().len(), 1, "holding it is not another reset"); + } + + #[test] + fn a_bridge_control_write_without_bit_6_leaves_the_secondary_alone() { + let r = rig(); + let log = Arc::new(Mutex::new(Vec::new())); + r.bridge.secondary().attach(at(2, 0), TestCard::logged("behind", bars(), 0, &log)); + r.root.config_write(at(3, 0), 0x3e, 2, 0x01); + assert!(log.lock().unwrap().is_empty()); + // RST# on the primary bus does reach behind the bridge. + r.root.reset(); + assert_eq!(*log.lock().unwrap(), ["behind cmd=0x0"]); + } + + #[test] + fn a_secondary_bus_saves_and_loads_with_its_parent() { + use crate::traits::Saveable; + let r = rig(); + child(&r, 2, 0, 0x8000_2000); + let v = r.root.save_state(); + let r2 = rig(); + child(&r2, 2, 0, 0x8000_6000); + r2.root.load_state(&v).unwrap(); + assert_eq!(r2.root.save_state(), v); + assert!(r2.root.decode(Space::Mem, 0x8000_2010).is_some() && r2.root.decode(Space::Mem, 0x8000_6010).is_none()); + } +} diff --git a/src/pci/testing.rs b/src/pci/testing.rs new file mode 100644 index 000000000..8fe35d60c --- /dev/null +++ b/src/pci/testing.rs @@ -0,0 +1,104 @@ +//! Test doubles for the bus: a function that records what it is asked, and +//! a host that records what it is told. + +use std::sync::{Arc, Mutex}; + +use super::{Bar, PciConfig, PciDma, PciDmaStatus, PciFunction, PciFunctionState, PciHost}; + +/// A function with the given BARs whose BAR 0 is a little-endian register +/// file, and which remembers the BAR calls it was given and, in a shared +/// log, when it was reset. It does not touch its interrupt line on reset, so +/// a test can see that the bus leaves the line alone. +pub(crate) struct TestCard { + pci: PciFunctionState, + mem: Mutex<[u8; 0x1000]>, + calls: Mutex>, + name: &'static str, + log: Arc>>, +} + +impl TestCard { + pub fn new(bars: [Bar; 6], pin: u8) -> Arc { + Self::logged("card", bars, pin, &Arc::default()) + } + + /// A card that writes `" cmd="` to `log` when it + /// is reset: the command shows whether its header was reset first. + pub fn logged(name: &'static str, bars: [Bar; 6], pin: u8, log: &Arc>>) -> Arc { + Arc::new(Self { + pci: PciFunctionState::new(PciConfig::new(0x1234, 0x5678, 0, 1, bars, pin)), + mem: Mutex::new([0; 0x1000]), + calls: Mutex::new(Vec::new()), + name, + log: log.clone(), + }) + } + + /// The register file. + pub fn mem(&self) -> Vec { + self.mem.lock().unwrap().to_vec() + } + + /// The (offset, width) of each BAR call so far. + pub fn calls(&self) -> Vec<(u32, usize)> { + self.calls.lock().unwrap().clone() + } +} + +impl PciFunction for TestCard { + fn state(&self) -> &PciFunctionState { + &self.pci + } + + fn bar_read(&self, _bar: usize, off: u32, width: usize) -> u32 { + self.calls.lock().unwrap().push((off, width)); + let m = self.mem.lock().unwrap(); + m[off as usize..off as usize + width].iter().rev().fold(0, |a, &b| a << 8 | b as u32) + } + + fn bar_write(&self, _bar: usize, off: u32, val: u32, width: usize) { + self.calls.lock().unwrap().push((off, width)); + self.mem.lock().unwrap()[off as usize..off as usize + width].copy_from_slice(&val.to_le_bytes()[..width]); + } + + + fn reset(&self) { + self.log.lock().unwrap().push(format!("{} cmd={:#x}", self.name, self.pci.config().command())); + } +} + +/// A DMA identity whose reads fill the buffer with the requester's number. +pub(crate) struct TestDma(pub u8); + +impl PciDma for TestDma { + fn dma_read(&self, _addr: u64, buf: &mut [u8]) -> PciDmaStatus { + buf.fill(self.0); + PciDmaStatus::Ok + } + + fn dma_write(&self, _addr: u64, _data: &[u8]) -> PciDmaStatus { + PciDmaStatus::Ok + } +} + +/// A host that logs each host IRQ change. +#[derive(Default)] +pub(crate) struct TestHost { + irqs: Mutex>, +} + +impl TestHost { + pub fn irqs(&self) -> Vec<(u8, bool)> { + self.irqs.lock().unwrap().clone() + } +} + +impl PciHost for TestHost { + fn dma_for(&self, requester: u8) -> Arc { + Arc::new(TestDma(requester)) + } + + fn set_irq(&self, irq: u8, level: bool) { + self.irqs.lock().unwrap().push((irq, level)); + } +} diff --git a/src/physical.rs b/src/physical.rs index d7e1ae193..aa0ab73fb 100644 --- a/src/physical.rs +++ b/src/physical.rs @@ -405,6 +405,12 @@ impl Physical { p } + /// How many devices the bus is keeping alive. + #[cfg(test)] + pub(crate) fn kept_len(&self) -> usize { + self.devices.lock().unwrap().len() + } + /// What answers wherever nothing else is mapped: the whole 32-bit space /// now, and any slot above it the first time the guest reaches it. pub fn set_default(&self, dev: Arc, name: &str) { diff --git a/src/ppmem/ppmem.rs b/src/ppmem/ppmem.rs index a30c0adb5..bd1dac80c 100644 --- a/src/ppmem/ppmem.rs +++ b/src/ppmem/ppmem.rs @@ -151,32 +151,14 @@ pub trait MappedMemory { /// a finer, multi-word bitmap. fn mapped_bitmap(&self) -> u64; - /// Redirect bitmap publication into the CPU's inline `ppmem_bitmap` field. - /// - /// Pass `MipsExecutor::ppmem_bitmap_ptr()`, queried once the executor is at - /// its final address. From then on every mapping change writes straight - /// through this pointer, so the CPU's RAM-vs-bus test is a load from the - /// object it is already executing out of — no indirection through - /// `PpMemSpace`, no atomic, no lock. - /// - /// Optional: until this is called the space publishes into a `u64` it owns - /// itself, so the sink is always valid and publication never branches. - /// - /// # Safety - /// - /// `ptr` must point to a live `u64` that outlives this `PpMemSpace`, and - /// must only ever be written from the CPU thread. Both hold for the CPU's - /// inline field: the executor never moves once inside its `Arc`, and the - /// only writer is the MEMCFG path on that same thread. - unsafe fn set_bitmap_sink(&self, ptr: *mut u64); - - /// A second publication target — see [`MappedMemory::set_bitmap_sink`]. + /// A second publication target, the cache's inline copy. /// /// Exists so the cache can hold the bitmap *inline* instead of chasing a /// pointer on every access. Both sinks receive every update. /// /// # Safety - /// Same contract as `set_bitmap_sink`. + /// `ptr` must point to a live `u64` that outlives this `PpMemSpace` and is + /// written only from the CPU thread. unsafe fn set_bitmap_sink2(&self, ptr: *mut u64); } @@ -656,11 +638,17 @@ impl BusDevice for Arc { // --------------------------------------------------------------------------- /// One live mapping inside the window, tracked so `clear_mappings` can undo it -/// and `mapped_bitmap` can be recomputed. +/// and `mapped_bitmap` can be recomputed. It also says where its bytes come +/// from, so a cached view can lay part of it again when a held page leaves: +/// `bank`'s object from `src`, or repeating every `period` bytes from `src` +/// into the period (0: no repeat). #[derive(Clone, Copy, Debug)] struct Mapping { at: u64, len: u64, + bank: usize, + src: u64, + period: u64, } /// The 4GB window that banks are mapped into, plus the banks themselves. @@ -692,7 +680,7 @@ pub struct PpMemSpace { /// `mapped_bitmap()`; written only via `state.sink`. own_bitmap: OwnBitmap, /// No lock: every path that touches this — `map_bank`, `map_alias`, - /// `clear_mappings`, `set_bitmap_sink` — runs on the CPU thread, reached + /// `clear_mappings`, `set_bitmap_sink2` — runs on the CPU thread, reached /// as CPU store → MC register write → `remap_banks`. No device thread ever /// enters this type. (`unsafe impl Sync` below records that contract.) state: UnsafeCell, @@ -711,8 +699,26 @@ struct SpaceState { /// path is one load instead of two — hence one sink per consumer rather /// than a shared indirection. Points at `own_bitmap` until claimed. sink2: *mut u64, - /// Further CPUs' inline copies (`PpMemSpace::add_bitmap_sink`). - more_sinks: Vec<*mut u64>, + /// The bitmap last published to the sinks above. + bits: u64, + /// Each CPU's cached view of memory: see [`PpMemSpace::add_cached_view`]. + views: Vec, + /// Ranges where a held page may stand with no memory mapped behind it: + /// see [`PpMemSpace::allow_unbacked_lines`]. + unbacked: Vec<(u64, u64)>, +} + +/// One CPU's cached view of memory: a window of its own, with every mapping +/// the shared window has, plus the pages its cache holds lines for, which +/// nobody else sees. +struct View { + space: AddrSpace, + /// Where this view's region bitmap is published: the CPU's inline copies. + sinks: Vec<*mut u64>, + /// The pages this view holds, with how many of the CPU's valid lines each + /// holds. Only its CPU changes them, at CACHE instructions, under the + /// contract below. + held: std::collections::BTreeMap, } /// The bitmap `PpMemSpace` publishes into before a CPU claims it. @@ -727,12 +733,15 @@ struct OwnBitmap(Box); // formality: // // * Every mutating entry point — `map_bank`, `map_alias`, `clear_mappings`, -// `set_bitmap_sink` — is reached only from `Physical::remap_banks`, which +// `set_bitmap_sink2` — is reached only from `Physical::remap_banks`, which // the MC invokes synchronously inside the CPU's store to MEMCFG. That is // always a CPU thread, and with several CPUs the others are parked for it // (`smp::World::stop_others`, wired where HEART's descriptors reach memory). // * Device threads (MC-DMA, seeq, VINO) reach RAM through `BusDevice` on the // individual `PpMemory` banks. They never touch `PpMemSpace`. +// * `hold_line_page`/`release_line_page` run on a CPU thread at a CACHE +// instruction and change only that CPU's own view. They read the mapping +// list and the bitmap, which only a remap changes, with the others parked. // * `mapped_bitmap()` reads the sink and is likewise CPU-thread-only; the // hot-path reader is the CPU's own inline `MipsCore::ppmem_bitmap` field, // which is a plain `u64` and never goes through here. @@ -769,7 +778,9 @@ impl PpMemSpace { mappings: Vec::new(), sink, sink2, - more_sinks: Vec::new(), + bits: 0, + views: Vec::new(), + unbacked: Vec::new(), }), }) } @@ -792,8 +803,8 @@ impl PpMemSpace { /// The `u64` the mapped-region bitmap is published into. /// - /// Never null: points at this space's own `u64` until a CPU claims it via - /// [`MappedMemory::set_bitmap_sink`], and at the CPU's inline field after. + /// Never null: this space's own `u64`. A CPU reads its cached view's + /// copy instead (`add_cached_view`). pub fn bitmap_ptr(&self) -> *const u64 { let st = unsafe { &*self.state.get() }; st.sink as *const u64 @@ -811,9 +822,11 @@ impl PpMemSpace { let b = &self.banks[bank]; assert!(src + size <= b.len() as u64, "ppmem: range [{src:#x}, +{size:#x}) escapes bank {bank}"); assert!(offset + size <= self.window_size(), "ppmem: [{offset:#x}, +{size:#x}) escapes the window"); + self.marks(false); unsafe { self.space.map(offset as usize, size as usize, b, src, Prot::ReadWrite)?; } + self.map_views(offset, size, b, src)?; #[cfg(feature = "jitv2")] { let (gen_off, gen_len, gen_src) = (offset / GEN_RATIO, size / GEN_RATIO, src / GEN_RATIO); @@ -827,7 +840,8 @@ impl PpMemSpace { } } let st = unsafe { &mut *self.state.get() }; - Self::record(st, offset, size); + Self::record(st, Mapping { at: offset, len: size, bank, src, period: 0 }); + self.marks(true); Ok(()) } @@ -867,37 +881,290 @@ impl PpMemSpace { unsafe { *state.sink = bits; *state.sink2 = bits; - for &s in &state.more_sinks { - *s = bits; - } + } + state.bits = bits; + for v in 0..state.views.len() { + Self::publish_view(state, v); + } + } + + /// Publish view `v`'s bitmap: the shared one, plus every region the view + /// holds a page in, which `hold_line_page` made safe to touch whole. + fn publish_view(state: &SpaceState, v: usize) { + let shift = window_shift(state.window); + let view = &state.views[v]; + let bits = view.held.keys().fold(state.bits, |b, &page| b | 1u64 << (page >> shift)); + for &s in &view.sinks { + unsafe { *s = bits }; } } - fn record(state: &mut SpaceState, at: u64, len: u64) { + fn record(state: &mut SpaceState, m: Mapping) { // Drop anything this mapping fully replaces, then record it. + let (at, len) = (m.at, m.len); state .mappings .retain(|m| !(m.at >= at && m.at + m.len <= at + len)); - state.mappings.push(Mapping { at, len }); + state.mappings.push(m); Self::recompute_bitmap(state); } } +/// Cached views. +/// +/// A CPU's cache and its compiled code read memory through ppmem's window, so +/// the window is that CPU's cached view of memory. Giving each CPU a window of +/// its own makes room for what the cache holds: a valid cache line's page can +/// become private memory in that CPU's view while the shared window, the +/// decode table and every other CPU go on seeing memory. Uncached accesses +/// bypass the caches (NEC VR10000 user's manual p. 75), and each CPU's cache +/// is its own. The views are not coherent: on a coherent system another +/// master's read would be answered from a dirty line, and here it reads +/// memory. +/// +/// Nothing on a load, store or fetch changes: the CPU reads its view through +/// the same base pointer and inline bitmap it read the shared window through. +/// The page is the host's mapping granule (4 KB on Linux and x86 macOS, +/// 16 KB on Apple silicon, 64 KB on Windows). impl PpMemSpace { - /// Publish the bitmap to one more `u64` as well: a second CPU's inline - /// copies. Same contract as [`MappedMemory::set_bitmap_sink`]. The value - /// arrives with the next remap. + /// Give a CPU a cached view of its own: a window with the same mappings + /// as this one, from now on, whose region bitmap is published to `sinks` + /// (the CPU's inline copies; null entries are skipped) instead of this + /// window's. Returns the view's number. Set up before the CPU runs. + pub fn add_cached_view(&self, sinks: &[*mut u64]) -> io::Result { + let st = unsafe { &mut *self.state.get() }; + let space = AddrSpace::reserve(st.window as usize)?; + let sinks = sinks.iter().copied().filter(|p| !p.is_null()).collect(); + st.views.push(View { space, sinks, held: Default::default() }); + let v = st.views.len() - 1; + for m in st.mappings.clone() { + self.lay(st, v, &m, m.at, m.at + m.len)?; + } + Self::publish_view(st, v); + Ok(v) + } + + /// The base of view `v`'s window. + pub fn view_base(&self, v: usize) -> *mut u8 { + unsafe { (&(*self.state.get()).views)[v].space.base() } + } + + /// The pages view `v` holds, in address order. + pub fn held_line_pages(&self, v: usize) -> Vec { + unsafe { (&(*self.state.get()).views)[v].held.keys().copied().collect() } + } + + /// A line of view `v`'s CPU at `pa` became valid: from now on the view + /// shows the page holding it as private memory, starting as a copy of + /// memory. Each call is one line, and [`Self::release_line_page`] gives + /// it back. Returns whether the page is newly held, or `None` when no + /// memory is mapped behind it. /// - /// # Safety - /// `ptr` must stay valid for as long as this space exists. - pub unsafe fn add_bitmap_sink(&self, ptr: *mut u64) { - unsafe { (*self.state.get()).more_sinks.push(ptr) }; + /// The region bitmap is all a CPU tests, so the page's whole region is + /// the view's while it stands, reading zeros where nothing is mapped. The + /// O2's POST holds lines for page 0 before CRIME maps the rest of its + /// region. + pub fn hold_line_page(&self, v: usize, pa: u64) -> Option { + let st = unsafe { &mut *self.state.get() }; + let gran = granularity() as u64; + let page = pa & !(gran - 1); + if !self.backs(page) && !st.unbacked.iter().any(|&(a, b)| page >= a && page + gran <= b) { + return None; + } + let shift = window_shift(st.window); + if let Some(n) = st.views[v].held.get_mut(&page) { + *n += 1; + return Some(false); + } + if !bitmap_maps_at(st.bits, shift, page) && !st.views[v].held.keys().any(|&p| p >> shift == page >> shift) { + self.scrub_region_gaps(st, v, page >> shift << shift, 1u64 << shift); + } + if !Self::held_anywhere(st, page) { + self.mark(st, page, true); + } + let view = &st.views[v]; + let at = unsafe { view.space.base().add(page as usize) }; + let mut bytes = vec![0u8; gran as usize]; + unsafe { + std::ptr::copy_nonoverlapping(at, bytes.as_mut_ptr(), bytes.len()); + let _ = view.space.scrub(page as usize, gran as usize); + std::ptr::copy_nonoverlapping(bytes.as_ptr(), at, bytes.len()); + } + st.views[v].held.insert(page, 1); + Self::publish_view(st, v); + Some(true) + } + + /// Whether memory is mapped behind the whole page holding `pa`. A page + /// [`Self::hold_line_page`] takes without it starts as zeros, for the + /// caller to fill. + pub fn backs(&self, pa: u64) -> bool { + let st = unsafe { &*self.state.get() }; + let gran = granularity() as u64; + let page = pa & !(gran - 1); + st.mappings.iter().any(|m| m.at <= page && m.at + m.len >= page + gran) + } + + /// Let held pages in `[start, end)` stand with no memory mapped behind + /// them: a PROM that keeps its stack and code in cache lines before it + /// has set up memory. + pub fn allow_unbacked_lines(&self, start: u64, end: u64) { + unsafe { (*self.state.get()).unbacked.push((start, end)) }; + } + + /// One line of [`Self::hold_line_page`]'s page is no longer valid. + /// Returns true when it was the page's last: the view shows memory there + /// again, and what the page held is dropped, as a line invalidated + /// without writeback is. + pub fn release_line_page(&self, v: usize, pa: u64) -> bool { + let st = unsafe { &mut *self.state.get() }; + let gran = granularity() as u64; + let page = pa & !(gran - 1); + let Some(n) = st.views[v].held.get_mut(&page) else { return false }; + *n -= 1; + if *n > 0 { + return false; + } + st.views[v].held.remove(&page); + if !Self::held_anywhere(st, page) { + self.mark(st, page, false); + } + let _ = unsafe { st.views[v].space.scrub(page as usize, gran as usize) }; + for m in st.mappings.iter().filter(|m| m.at < page + gran && m.at + m.len > page) { + let _ = self.lay(st, v, m, page.max(m.at), (page + gran).min(m.at + m.len)); + } + Self::publish_view(st, v); + true + } + + fn held_anywhere(st: &SpaceState, page: u64) -> bool { + st.views.iter().any(|view| view.held.contains_key(&page)) + } + + /// A page some view holds carries one `GEN_HELD` in its generation + /// counters while memory backs it, so jitv2 does not compile it. + #[cfg_attr(not(feature = "jitv2"), allow(unused_variables))] + fn mark(&self, st: &SpaceState, page: u64, on: bool) { + #[cfg(feature = "jitv2")] + { + use crate::cpu::jitv2::jitv2::{GEN_HELD, PAGE_SIZE}; + let end = page + granularity() as u64; + if !st.mappings.iter().any(|m| m.at <= page && m.at + m.len >= end) { + return; + } + let d = if on { GEN_HELD } else { 1u64.wrapping_sub(GEN_HELD) }; + let gens = self.gen_space.base() as *const AtomicU64; + for a in (page..end).step_by(PAGE_SIZE as usize) { + unsafe { (*gens.add((a / PAGE_SIZE as u64) as usize)).fetch_add(d, Ordering::Release) }; + } + } + } + + /// Every held page's mark, off before a mapping change and on after. + fn marks(&self, on: bool) { + let st = unsafe { &*self.state.get() }; + let pages: std::collections::BTreeSet = st.views.iter().flat_map(|v| v.held.keys().copied()).collect(); + for page in pages { + self.mark(st, page, on); + } + } + + /// Make the parts of `[r, r + len)` that nothing maps private zeros in + /// view `v`, and give them generation counters where none are mapped, so + /// the region can be marked whole in the view's bitmap: compiled stores + /// there bump a counter. + fn scrub_region_gaps(&self, st: &SpaceState, v: usize, r: u64, len: u64) { + let end = r + len; + let mut at = r; + while at < end { + if let Some(m) = st.mappings.iter().find(|m| m.at <= at && m.at + m.len > at) { + at = m.at + m.len; + continue; + } + let next = st.mappings.iter().filter(|m| m.at > at && m.at < end).map(|m| m.at).min().unwrap_or(end); + let _ = unsafe { st.views[v].space.scrub(at as usize, (next - at) as usize) }; + #[cfg(feature = "jitv2")] + { + let g = granularity() as u64; + let (gs, ge) = ((at / GEN_RATIO) & !(g - 1), (next / GEN_RATIO).div_ceil(g) * g); + let touches = |m: &Mapping| m.at < ge * GEN_RATIO && m.at + m.len > gs * GEN_RATIO; + if !st.mappings.iter().any(touches) { + let _ = unsafe { self.gen_space.scrub(gs as usize, (ge - gs) as usize) }; + } + } + at = next; + } + } + + /// A mapping just laid in the shared window goes into every view too, + /// under the pages each holds. + fn map_views(&self, at: u64, len: u64, b: &SharedMem, src: u64) -> io::Result<()> { + let st = unsafe { &*self.state.get() }; + for view in &st.views { + let saved = Self::save_held(view, at, len); + unsafe { view.space.map(at as usize, len as usize, b, src, Prot::ReadWrite)? }; + Self::put_held(view, saved); + } + Ok(()) + } + + /// Lay `[from, to)` of mapping `m` in view `v`. Data only: the views + /// share the window's generation counters. + fn lay(&self, st: &SpaceState, v: usize, m: &Mapping, from: u64, to: u64) -> io::Result<()> { + let b = &self.banks[m.bank]; + let mut at = from; + while at < to { + let rel = at - m.at; + let (src, run) = if m.period == 0 { + (m.src + rel, to - at) + } else { + let o = (m.src + rel) % m.period; + (o, (m.period - o).min(to - at)) + }; + unsafe { st.views[v].space.map(at as usize, run as usize, b, src, Prot::ReadWrite)? }; + at += run; + } + Ok(()) + } + + /// The bytes of each page `view` holds inside `[at, at + len)`, before + /// a mapping change replaces them. + fn save_held(view: &View, at: u64, len: u64) -> Vec<(u64, Vec)> { + let gran = granularity(); + let base = view.space.base(); + view.held.keys() + .filter(|&&p| p < at + len && p + gran as u64 > at) + .map(|&p| { + let mut b = vec![0u8; gran]; + unsafe { std::ptr::copy_nonoverlapping(base.add(p as usize), b.as_mut_ptr(), gran) }; + (p, b) + }) + .collect() + } + + /// Put [`Self::save_held`]'s pages back, private again. + fn put_held(view: &View, saved: Vec<(u64, Vec)>) { + let base = view.space.base(); + for (page, bytes) in saved { + let _ = unsafe { view.space.scrub(page as usize, bytes.len()) }; + unsafe { std::ptr::copy_nonoverlapping(bytes.as_ptr(), base.add(page as usize), bytes.len()) }; + } } } impl MappedMemory for PpMemSpace { fn clear_mappings(&self) { + self.marks(false); let st = unsafe { &mut *self.state.get() }; + // Views keep the pages they hold. Scrubbed, not left mapped, so a + // view never shows memory the shared window no longer maps. + for view in &st.views { + let saved = Self::save_held(view, 0, st.window); + for m in &st.mappings { + let _ = unsafe { view.space.scrub(m.at as usize, m.len as usize) }; + } + Self::put_held(view, saved); + } for m in std::mem::take(&mut st.mappings) { // Best-effort: a failure here leaves the range mapped, which is // still safe — it just isn't reverted. @@ -915,16 +1182,11 @@ impl MappedMemory for PpMemSpace { } } } - unsafe { - *st.sink = 0; - *st.sink2 = 0; - for &s in &st.more_sinks { - *s = 0; - } - } + Self::recompute_bitmap(st); } fn map_bank(&self, bank: usize, offset: u64, size: u64, period: u64) -> io::Result<()> { + self.marks(false); let b = &self.banks[bank]; let bank_size = b.len() as u64; assert!(size > 0, "ppmem: map_bank with zero size"); @@ -950,6 +1212,7 @@ impl MappedMemory for PpMemSpace { self.space .map(at as usize, chunk as usize, b, 0, Prot::ReadWrite)?; } + self.map_views(at, chunk, b, 0)?; // Lockstep: the counters for this mirror land at the same offset // scaled by GEN_RATIO, from the same bank's gen object. Every // mirror therefore shares one physical counter page with the @@ -975,11 +1238,13 @@ impl MappedMemory for PpMemSpace { } let st = unsafe { &mut *self.state.get() }; - Self::record(st, offset, size); + Self::record(st, Mapping { at: offset, len: size, bank, src: 0, period }); + self.marks(true); Ok(()) } fn map_alias(&self, bank: usize, offset: u64, size: u64) -> io::Result<()> { + self.marks(false); let b = &self.banks[bank]; assert!( size <= b.len() as u64, @@ -994,6 +1259,7 @@ impl MappedMemory for PpMemSpace { self.space .map(offset as usize, size as usize, b, 0, Prot::ReadWrite)?; } + self.map_views(offset, size, b, 0)?; // Alias the counters too — this is NOT optional. // // An earlier version skipped the gen mapping whenever the alias's gen @@ -1045,7 +1311,8 @@ impl MappedMemory for PpMemSpace { } } let st = unsafe { &mut *self.state.get() }; - Self::record(st, offset, size); + Self::record(st, Mapping { at: offset, len: size, bank, src: 0, period: 0 }); + self.marks(true); Ok(()) } @@ -1058,15 +1325,6 @@ impl MappedMemory for PpMemSpace { self.gen_space.base() as *mut AtomicU64 } - unsafe fn set_bitmap_sink(&self, ptr: *mut u64) { - let st = unsafe { &mut *self.state.get() }; - // Carry the current value across: the boot-time remap runs before the - // CPU thread starts, so banks are typically already mapped by now. - let current = unsafe { *st.sink }; - st.sink = ptr; - unsafe { *ptr = current }; - } - unsafe fn set_bitmap_sink2(&self, ptr: *mut u64) { let st = unsafe { &mut *self.state.get() }; let current = unsafe { *st.sink }; @@ -1354,33 +1612,6 @@ mod tests { assert_eq!(space.mapped_bitmap() & 1, 0, "partial region must read unmapped"); } - /// The bitmap must land in whatever `u64` the CPU handed over, so the hot - /// path reads it as a plain inline field. - #[test] - fn bitmap_publishes_into_the_cpu_field() { - let (space, _banks) = PpMemSpace::with_bank_sizes(&[64]).unwrap(); - - // Stand in for MipsCore.ppmem_bitmap. Boxed so its address is stable. - let cpu_field = Box::new(0u64); - let ptr = &*cpu_field as *const u64 as *mut u64; - - // Map before registering: the boot-time remap runs before the CPU - // thread exists, so the current value must carry across. - space.map_bank(0, 0x0800_0000, 64 * MB as u64, 64 * MB as u64).unwrap(); - let expected = space.mapped_bitmap(); - assert_ne!(expected, 0); - - unsafe { space.set_bitmap_sink(ptr) }; - assert_eq!(*cpu_field, expected, "existing bitmap not carried to the CPU field"); - - // Subsequent changes must publish straight through. - space.map_bank(0, 0x1000_0000, 64 * MB as u64, 64 * MB as u64).unwrap(); - assert_eq!(*cpu_field, space.mapped_bitmap(), "later remap did not publish"); - - space.clear_mappings(); - assert_eq!(*cpu_field, 0, "clear_mappings did not publish"); - } - /// The bit the CPU tests must be the one covering that physical address. #[test] fn cpu_bitmap_test_selects_the_right_region() { @@ -1696,4 +1927,166 @@ mod tests { let gen_bytes = min_bank / GEN_RATIO as usize; assert_eq!(gen_bytes, 16 * 1024, "8MB bank => 16KB of counters"); } + + // ---- cached views ---- + + /// A CPU's view of `space`, publishing into a boxed stand-in for its + /// inline bitmap field. + fn view_of(space: &PpMemSpace) -> (usize, Box) { + let field = Box::new(0u64); + let v = space.add_cached_view(&[&*field as *const u64 as *mut u64]).unwrap(); + (v, field) + } + + fn word(base: *mut u8, at: u64) -> u32 { + unsafe { *(base.add(at as usize) as *const u32) } + } + + fn set_word(base: *mut u8, at: u64, w: u32) { + unsafe { *(base.add(at as usize) as *mut u32) = w }; + } + + /// A view shows every mapping the window has, laid before or after it + /// was made, and publishes the same bitmap to its own sink. + #[test] + fn a_view_follows_the_windows_mappings() { + let (space, banks) = PpMemSpace::with_bank_sizes(&[64, 64]).unwrap(); + space.map_bank(0, 0, 64 * MB as u64, 64 * MB as u64).unwrap(); + let (v, field) = view_of(&space); + space.map_alias(1, 0x0800_0000, 64 * MB as u64).unwrap(); + banks[0].write32(0x40, 0x1111_1111); + banks[1].write32(0x80, 0x2222_2222); + assert_eq!(word(space.view_base(v), 0x40), 0x1111_1111); + assert_eq!(word(space.view_base(v), 0x0800_0080), 0x2222_2222); + assert_eq!(*field, space.mapped_bitmap()); + set_word(space.view_base(v), 0x44, 0x3333_3333); + assert_eq!(banks[0].read32(0x44).data, 0x3333_3333, "an unheld page is memory itself"); + space.clear_mappings(); + assert_eq!(*field, 0); + } + + /// A held page is the view's own: what is written there stays there, the + /// window and memory keep their bytes, and releasing it drops the page. + #[test] + fn a_held_page_is_private_to_its_view_until_released() { + let (space, banks) = PpMemSpace::with_bank_sizes(&[64]).unwrap(); + space.map_bank(0, 0, 64 * MB as u64, 64 * MB as u64).unwrap(); + let (v, _field) = view_of(&space); + banks[0].write32(0x2040, 0x1234_5678); + assert_eq!(space.hold_line_page(v, 0x2040), Some(true)); + assert_eq!(space.hold_line_page(v, 0x2080), Some(false), "a second line of the page"); + let base = space.view_base(v); + assert_eq!(word(base, 0x2040), 0x1234_5678, "it starts as memory"); + set_word(base, 0x2040, 0xaaaa_aaaa); + banks[0].write32(0x2044, 0x5555_5555); + assert_eq!(banks[0].read32(0x2040).data, 0x1234_5678); + assert_eq!(word(space.window_base(), 0x2040), 0x1234_5678); + assert_eq!(word(base, 0x2044), 0, "the page does not see later memory writes"); + assert!(!space.release_line_page(v, 0x2040), "one line left"); + assert_eq!(space.held_line_pages(v), vec![0x2000]); + assert!(space.release_line_page(v, 0x2080)); + assert!(space.held_line_pages(v).is_empty()); + assert_eq!(word(base, 0x2040), 0x1234_5678, "memory again"); + assert_eq!(word(base, 0x2044), 0x5555_5555); + } + + /// Two CPUs holding one page each see their own bytes. + #[test] + fn two_views_hold_one_page_apart() { + let (space, _banks) = PpMemSpace::with_bank_sizes(&[64]).unwrap(); + space.map_bank(0, 0, 64 * MB as u64, 64 * MB as u64).unwrap(); + let (a, _fa) = view_of(&space); + let (b, _fb) = view_of(&space); + space.hold_line_page(a, 0x10_0000); + space.hold_line_page(b, 0x10_0000); + set_word(space.view_base(a), 0x10_0040, 0xaaaa_aaaa); + set_word(space.view_base(b), 0x10_0040, 0xbbbb_bbbb); + assert_eq!(word(space.view_base(a), 0x10_0040), 0xaaaa_aaaa); + assert_eq!(word(space.view_base(b), 0x10_0040), 0xbbbb_bbbb); + assert_eq!(word(space.window_base(), 0x10_0040), 0); + } + + /// A remap lays memory out again under every view and must not take a + /// held page with it. + #[test] + fn a_remap_keeps_held_pages() { + let (space, banks) = PpMemSpace::with_bank_sizes(&[64]).unwrap(); + space.map_bank(0, 0, 64 * MB as u64, 64 * MB as u64).unwrap(); + let (v, field) = view_of(&space); + space.hold_line_page(v, 0x3000); + set_word(space.view_base(v), 0x3010, 0x7777_7777); + space.clear_mappings(); + assert_eq!(*field, 1, "the region the page stands in stays the view's"); + space.map_bank(0, 0, 64 * MB as u64, 32 * MB as u64).unwrap(); + assert_eq!(word(space.view_base(v), 0x3010), 0x7777_7777); + assert_eq!(banks[0].read32(0x3010).data, 0); + space.release_line_page(v, 0x3000); + banks[0].write32(0x3010, 0x9999_9999); + assert_eq!(word(space.view_base(v), 0x3010), 0x9999_9999); + assert_eq!(word(space.view_base(v), 0x200_3010), 0x9999_9999, "laid again with the bank's period"); + } + + /// A page with no memory behind it cannot be held. One in a region the + /// window maps only in part makes the region the view's, reading zeros + /// where nothing is mapped, until the page goes. + #[test] + fn a_page_needs_memory_and_takes_its_region() { + let (space, banks) = PpMemSpace::with_bank_sizes(&[32]).unwrap(); + let (v, field) = view_of(&space); + assert_eq!(space.hold_line_page(v, 0x10_0000), None); + assert_eq!(space.hold_line_page(v, space.window_size()), None, "outside the window"); + space.map_bank(0, 0, 32 * MB as u64, 32 * MB as u64).unwrap(); + assert_eq!((space.mapped_bitmap(), *field), (0, 0), "a 32 MB bank leaves its 64 MB region to the bus"); + assert_eq!(space.hold_line_page(v, 0x40 * MB as u64), None, "nothing behind it"); + banks[0].write32(0x10_0040, 0x600d); + assert_eq!(space.hold_line_page(v, 0x10_0040), Some(true)); + assert_eq!(*field, 1); + assert_eq!(space.mapped_bitmap(), 0, "the shared window is unchanged"); + assert_eq!(word(space.view_base(v), 0x10_0040), 0x600d, "the page starts as memory"); + assert_eq!(word(space.view_base(v), 0x3ff_fffc), 0, "the rest of the region is safe to touch"); + space.release_line_page(v, 0x10_0040); + assert_eq!(*field, 0); + } + + /// A held page's counter carries one `GEN_HELD` however many views hold + /// it, and the mark follows the page to the counter that backs it after + /// a remap, leaving none on the old one. + #[cfg(feature = "jitv2")] + #[test] + fn the_held_mark_follows_the_page() { + use crate::cpu::jitv2::jitv2::GEN_HELD; + let (space, banks) = PpMemSpace::with_bank_sizes(&[64, 64]).unwrap(); + space.map_bank(0, 0, 64 * MB as u64, 64 * MB as u64).unwrap(); + let ((a, _fa), (b, _fb)) = (view_of(&space), view_of(&space)); + let gen = |g: *const AtomicU64| unsafe { (*g).load(Ordering::Relaxed) }; + let at = || unsafe { space.gen_window_base().add(0x2000 >> 12) as *const AtomicU64 }; + space.hold_line_page(a, 0x2000); + space.hold_line_page(b, 0x2040); + assert_eq!(gen(at()) / GEN_HELD, 1, "one mark for two CPUs"); + space.clear_mappings(); + space.map_bank(1, 0, 64 * MB as u64, 64 * MB as u64).unwrap(); + assert_eq!(gen(at()) / GEN_HELD, 1, "on the counter that backs the page now"); + assert_eq!(gen(banks[0].gen_ptr(0x2000)) / GEN_HELD, 0, "and off the old one"); + space.release_line_page(a, 0x2000); + assert_eq!(gen(at()) / GEN_HELD, 1); + space.release_line_page(b, 0x2040); + assert_eq!(gen(at()) / GEN_HELD, 0); + } + + /// With no memory behind it a page is refused unless the machine allows + /// lines there, and then its region is the view's until the page goes. + #[test] + fn an_unbacked_page_needs_the_machines_leave() { + let (space, _banks) = PpMemSpace::with_bank_sizes(&[64]).unwrap(); + let (v, field) = view_of(&space); + assert_eq!(space.hold_line_page(v, 0x10_0000), None); + space.allow_unbacked_lines(0, 64 * MB as u64); + assert!(!space.backs(0x10_0000)); + assert_eq!(space.hold_line_page(v, 0x10_0000), Some(true)); + assert_eq!(*field, 1); + assert_eq!(space.mapped_bitmap(), 0, "the shared window is unchanged"); + assert_eq!(word(space.view_base(v), 0x3ff_fffc), 0, "the region is safe to touch"); + space.release_line_page(v, 0x10_0000); + assert_eq!(*field, 0); + } } diff --git a/src/scsi.rs b/src/scsi.rs index bfb2ffda8..eb82e3347 100644 --- a/src/scsi.rs +++ b/src/scsi.rs @@ -162,6 +162,7 @@ pub struct ScsiDevice { kind: DeviceKind, buffer: Vec, // Internal buffer for WRITE_BUFFER/READ_BUFFER commands pending_sense: [u8; 18], // Stored sense data, served on REQUEST_SENSE + pending_attention: Option, /// Physical sector size: always 2048 for CD-ROM (Yellow Book), 512 for HDD. /// Used for TOC lead-out LBA (CD spec requires TOC in physical sectors). phys_block_size: u64, @@ -222,6 +223,7 @@ impl ScsiDevice { kind: if is_cdrom { DeviceKind::Cdrom } else { DeviceKind::Disk }, buffer: vec![0u8; SCSI_BUFFER_SIZE], pending_sense: [0u8; 18], + pending_attention: None, phys_block_size: if is_cdrom { CDROM_SECTOR } else { 512 }, // CD-ROM drives default to 2048-byte logical blocks (Sony CDU-76S behaviour). // HDD defaults to 512. dksc switches CD-ROM to 512 for EFS, back to 2048 for ISO. @@ -242,6 +244,7 @@ impl ScsiDevice { kind: DeviceKind::Cdrom, buffer: vec![0u8; SCSI_BUFFER_SIZE], pending_sense: [0u8; 18], + pending_attention: None, phys_block_size: CDROM_SECTOR, logical_block_size: 2048, identity, @@ -326,6 +329,10 @@ impl ScsiDevice { ScsiResponse { status: 0x02, data: vec![] } } + fn take_attention(&mut self) -> Option { + self.pending_attention.take().or_else(|| self.media.take_changed().then_some(0x28)) + } + fn exec_test_unit_ready(&mut self, _cdb: &[u8]) -> Result { if self.media.take_changed() { // Sense key 0x06 UNIT_ATTENTION, ASC 0x28 "Not Ready to Ready Transition / Medium Changed" @@ -1118,11 +1125,10 @@ impl ScsiTarget for ScsiDevice { } // Unit attention fires before any other command (except REQUEST_SENSE/INQUIRY). - if req.cdb[0] != scsi_cmd::REQUEST_SENSE - && req.cdb[0] != scsi_cmd::INQUIRY - && self.media.take_changed() - { - return Ok(self.check_condition(0x06, 0x28, 0x00)); + if req.cdb[0] != scsi_cmd::REQUEST_SENSE && req.cdb[0] != scsi_cmd::INQUIRY { + if let Some(asc) = self.take_attention() { + return Ok(self.check_condition(0x06, asc, 0x00)); + } } let mut response = match req.cdb[0] { @@ -1268,6 +1274,12 @@ impl ScsiTarget for ScsiDevice { fn block_size(&self) -> Option { Some(self.logical_block_size) } + fn bus_reset(&mut self) { + self.pending_sense = [0; 18]; + self.pending_sense[0] = 0x70; + self.pending_attention = Some(0x29); + } + fn reset_block_size(&mut self, cdrom_512: bool) { if cdrom_512 && self.is_cdrom() { self.logical_block_size = 512; @@ -1867,12 +1879,38 @@ mod tests { cd.changer().unwrap().load(&chd.display().to_string()).unwrap(); assert!(cd.has_media()); assert_eq!(cd.size(), 16 * 2048, "the CHD's size, not the file's"); - // The medium change is reported once, then the disc reads. + // A reset is reported before a media change that was already pending. + crate::scsi::ScsiTarget::bus_reset(&mut cd); + let reset = cd.request(&request(&[0, 0, 0, 0, 0, 0], None)).unwrap(); + assert_eq!(reset.status, 0x02, "unit attention after the reset"); + let reset_sense = cd.request(&request(&[0x03, 0, 0, 0, 18, 0], None)).unwrap(); + assert_eq!(reset_sense.data[12], 0x29, "reset attention comes first"); + let tur = cd.request(&request(&[0, 0, 0, 0, 0, 0], None)).unwrap(); - assert_eq!(tur.status, 0x02, "unit attention after the change"); + assert_eq!(tur.status, 0x02, "the pending media change follows"); + let sense = cd.request(&request(&[0x03, 0, 0, 0, 18, 0], None)).unwrap(); + assert_eq!(sense.data[12], 0x28, "media change keeps its ASC"); let read = cd.request(&request(&[0x28, 0, 0, 0, 0, 3, 0, 0, 1, 0], None)).unwrap(); assert_eq!(read.status, 0x00); assert_eq!(read.data, vec![0x42u8; 2048]); let _ = std::fs::remove_dir_all(dir); } + #[test] + fn bus_reset_reports_unit_attention_once_and_keeps_block_size() { + let mut dev = cdrom(); + dev.logical_block_size = 512; + dev.set_sense(0x05, 0x20, 0x00); + crate::scsi::ScsiTarget::bus_reset(&mut dev); + + let old_sense = run(&mut dev, &[0x03, 0, 0, 0, 18, 0]); + assert_eq!((old_sense.data[2], old_sense.data[12]), (0, 0), "reset clears previous sense"); + let response = run(&mut dev, &[0x00, 0, 0, 0, 0, 0]); + assert_eq!(response.status, 0x02, "a bus reset arms unit attention"); + let sense = run(&mut dev, &[0x03, 0, 0, 0, 18, 0]); + assert_eq!(sense.data[12], 0x29, "the reset attention has the reset ASC"); + + let response = run(&mut dev, &[0x00, 0, 0, 0, 0, 0]); + assert_eq!(response.status, 0x00, "the attention is reported once"); + assert_eq!(dev.block_sizes().1, 512, "bus reset leaves the selected block size alone"); + } } diff --git a/src/scsi/bus.rs b/src/scsi/bus.rs index 4283d763f..cb978d19a 100644 --- a/src/scsi/bus.rs +++ b/src/scsi/bus.rs @@ -51,8 +51,15 @@ pub trait ScsiTarget: Send { /// The logical block size, for a target that has one. fn block_size(&self) -> Option { None } - /// A controller's bus reset: block size back to what the controller - /// leaves, with a CD-ROM at 512 bytes if `cdrom_512`. + /// A SCSI bus reset reached this target, which owes the initiator a + /// unit attention on its next command. The ISP12160 calls this when the + /// guest resets either of its buses; the WD33C93A, AIC-7880 and ISP1040 + /// models never told their targets about a reset and still do not. + fn bus_reset(&mut self) {} + + /// The AIC-7880's bus reset: block size back to what that controller + /// leaves, with a CD-ROM at 512 bytes if `cdrom_512`. Only the AIC-7880 + /// calls this. fn reset_block_size(&mut self, _cdrom_512: bool) {} /// Put the mode parameters a guest can change back to what the target diff --git a/src/snapshot.rs b/src/snapshot.rs index f38835381..186c4877c 100644 --- a/src/snapshot.rs +++ b/src/snapshot.rs @@ -49,7 +49,13 @@ use toml::Value; /// changed shape: the serial DMA, the Indy's SCC and the rest are as before. /// Octane snapshots start at v5, and an Octane reads only snapshots of the /// schema it was built with, so the next bump refuses every older one. -pub const SCHEMA_VERSION: u32 = 5; +/// +/// v5 → v6: the PCI bus saves its own state (each function's header, the pin +/// and wire levels) where MACE and the Bridge each saved headers and pin +/// levels in their own layout. A v5 reader would load none of it and leave +/// every BAR unassigned under a running kernel, and a v6 reader does the +/// same with a v5 snapshot, so both machines refuse the other schema. +pub const SCHEMA_VERSION: u32 = 6; const MANIFEST_FILE: &str = "snapshot.toml"; @@ -127,6 +133,16 @@ pub fn enabled_features() -> Vec { f } +/// Refuse a snapshot of any schema but the one this build writes. The +/// Octane and the O2 read only their own schema; the Indy's loader reads +/// older ones itself. +pub fn check_schema(name: &str, found: u32) -> Result<(), String> { + if found != SCHEMA_VERSION { + return Err(format!("snapshot {name} is schema {found}; this build reads schema {SCHEMA_VERSION}")); + } + Ok(()) +} + /// Check that a snapshot fits this build, CPU and disks: the same build /// features, CPU model and cache model, and every disk it recorded configured /// at the same size. `IRIS_SNAPSHOT_SKIP_CHECK=1` makes a mismatch a warning. @@ -756,6 +772,16 @@ impl StateTable { mod tests { use super::*; + /// A snapshot of another schema is refused with both versions named. + #[test] + fn check_schema_names_both_versions() { + let older = SCHEMA_VERSION - 1; + let e = check_schema("old", older).unwrap_err(); + assert!(e.contains(&format!("schema {older}")) && e.contains(&format!("schema {SCHEMA_VERSION}")), "{e}"); + assert!(check_schema("newer", SCHEMA_VERSION + 1).is_err()); + assert!(check_schema("current", SCHEMA_VERSION).is_ok()); + } + fn unique_tmp_dir(tag: &str) -> PathBuf { let nanos = std::time::SystemTime::now() .duration_since(std::time::UNIX_EPOCH) diff --git a/src/workstation.rs b/src/workstation.rs index 2bd5798da..1b6b7fae9 100644 --- a/src/workstation.rs +++ b/src/workstation.rs @@ -40,6 +40,7 @@ use crate::traits::{Device, MachineEvent}; pub fn build(cfg: MachineConfig) -> Result, String> { crate::machine::set_emulator_profile(cfg.machine.profile); match cfg.machine.profile { + crate::config::MachineProfile::FuelIp34 => Ok(Box::new(crate::ip35_machine::FuelMachine::new(&cfg)?)), // The O2 is its own machine (CRIME, MACE, GBE), not an MC/HPC3 one. crate::config::MachineProfile::O2Ip32 => Ok(Box::new(crate::ip32_machine::O2Machine::new(&cfg)?)), // So is the Octane (HEART, Xbow, Bridge).