Repository navigation
ip27: clean up the IP27 machines, and fixes found bringing up the Onyx2 - #481
Draft
jamesbraid wants to merge 28 commits into
Draft
jamesbraid wants to merge 28 commits into
jamesbraid wants to merge 28 commits into
Conversation
The R10000 family puts a two-bit uncached attribute beside the address of every uncached reference. An unmapped one takes it from XKPHYS bits 58:57, which already reach the board in those bits; a mapped one takes it from EntryLo bits 63:62, which the TLB stored and then dropped when it made the physical address. Every mapped uncached reference therefore went out with attribute 0. The Onyx2's Hub picks node memory, its own registers or a crossbar widget by that attribute. IRIX maps the InfiniteReality pipe's command page into the kernel and into the graphics start-up and X server as an uncached accelerated page with attribute 1, I/O. With attribute 0 the stores landed in the Hub-special view at node memory past the fitted 128 MiB and vanished: the kernel's pipe start never reached the pipe, and the driver gave up with "IR0: mopup timeout" and death reason 17. A TLB entry with cache algorithm 2 (uncached) or 7 (uncached accelerated) now carries its attribute in bits 58:57 of the physical address it gives, the same place an unmapped reference carries it. Cached entries are unchanged. The attribute is folded in when the entry is written, so translation does no extra work. The other processors cannot store those EntryLo bits, and the boards that do not decode the attribute cut it off with the executor's physical mask.
A device that makes the processor wait answers a read busy and gets it
again. The Hub's processor path took any answer but OK for a failed read
and latched its address in the error status, so a pipe that answers
busy while its command ring drains left "Read Access Error" behind, and
IRIX panicked on it at start-up ("Invalid RRB in error stack") when it
read the Hub's error state. Only a failed read is recorded now.
The Onyx2 read its node count, flash and controller store from [origin2000], so its guide had to explain that it was configured under another product's name, and the Origin 200 and Origin 2000 each had a section struct with the same keys. No other machine does either: [o2], [octane], and the Fuel and Tezro each have one section named for the product. [origin200], [origin2000] and the new [onyx2] now deserialise into one Ip27Section, and MachineConfig::ip27_section returns the one the profile reads, so board code no longer names another product's table. validate() refuses a table that is not the profile's own, naming the one to rename it to, which turns an Onyx2 file that still says [origin2000] into a clear error rather than silently ignored keys. It also refuses nodes on the Origin 200, which has one node board. The keys landed only days ago, so the rename is cheapest now, before users have files. FEATURES.md gains the missing [origin200] table and an onyx2.toml.example is added.
The node PROM copies the I/O PROM from the BaseIO's flash at "Loading BASEIO prom" and, finding nothing, runs its own reduced copy, which has no graphics console: with console set to g it prints "Cannot open /dev/graphics/textport" and stays on the serial port. io_prom, in [origin2000] or [onyx2], names the image the IRIX 6.5.30 media ship beside the node PROM. It is fitted as a 1 MiB x16 part on the Bridge's flash port that answers reset and autoselect with the codes the loader accepts, and refuses program and erase. Without the key nothing is fitted, as before. The Origin 200 has no BaseIO card and refuses the key.
The three SN0 machines were spelled out as an or-pattern about ten times in config.rs and three times in machine.rs, which also repeated the same word list for the Origin 2000 and the Onyx2. MachineProfile::is_ip27 says it once, ip27_name gives the product's name for a sentence, and the machine.rs lists and tests walk MachineProfile::EVERY, so a fourth product cannot be left out of one of them. Several messages named the wrong machine along the way. The DaynaPort refusal said "not available on the O2" for all three SN0 machines. network.mode = "off" listed "the Origin 200 and 2000" and left out the Onyx2. The second-SCSI-controller refusal listed only the Indigo2 and the O2, although the Octane and the SN0 machines also have two; it now derives the list from scsi_controllers.
The Origin 200, Origin 2000 and Onyx2 test modules each opened with the same 60 lines: a scratch directory, a PROM stand-in and a base configuration that differed only in the profile, plus a copy of big_stack that display_stop_tests already provides. They now share one test_support module. The configuration also bound three listeners on port 0 and dropped them before the machine started, so another test thread could take a port between the drop and its use. The serial backend binds the port it is given and reports the one it got, so the serial ports are 0 and the machine finds its own free ones. The monitor port stays unset, since no test starts a machine. Scratch directories take their product as a prefix, because the three modules now share one name space inside one process.
The machine module and the Hub's own documentation both called Hub::inbound and Hub::interrupt_target "the smallest adapter the Bridge needs, meant to be replaced by the shared host interface". That interface, xtalk::HostPort, landed in the same series and takes exactly those two parts. Hub::host_port(widget) builds one, with the Hub's 128 interrupt numbers, and all four places that wired a Bridge to a Hub by hand use it. The two stale paragraphs go. The Origin 2000's Bridge sits behind a Crossbow but was built with Upstream::single, so its transfers and interrupt packets ignored the Crossbow's link. It now takes xbow.upstream, as the Onyx2 does, and the port's link gates them. The model has no way to take a fitted port down (a link reset brings it back, and only a snapshot restore clears an alive bit), and the PROM is not known to try, so nothing a guest can do shows the difference yet; the new test pins the wiring instead. A Bridge on the first node's Crossbow reaches that node's Hub at widget 9, and a packet naming the second node's widget 10 finds no host.
The Octane, the Origin 2000 and the Onyx2 each put a Bridge behind a Crossbow, and each repeated the same wiring: tell the Bridge which widgets have a live link, and reset the Bridge when the Crossbow resets its link. Xbow::serve_bridge does both for the port the Bridge sits on. The Octane's behaviour is unchanged; its wiring moved from two places to one call. The existing Octane tests that drive a link reset (ip30::tests and the Crossbow's link-reset tests) pass as they were, and a new Crossbow test shows a reset of the Bridge's port resets the Bridge while a reset of another port does not.
Onyx2Deskside::specs and Origin2000::specs were the same 25 lines, apart from the slot identity, and each file declared its own widget_hub, slot_bits and hub_host and built the BaseIO's Bridge the same way. The Indy and Indigo2 are one machine with the product as data; the IP27 machines already had one Ip27Machine over a board, so the desksides become one board, Deskside, and the product a DesksideLayout read once when the machine is built: each node slot's Hub widget and slot identity, the BaseIO's widget, board record and keyboard ports, whether the node boards share one Crossbow, and the graphics slot's widget if the product has one. Nothing tests the product on an access. The two topologies stay two build-time functions, wire_per_node for the Origin 2000's Crossbow on each node board and wire_shared for the Onyx2's one Crossbow; they and specs, controller and the Bridge builder share everything else. The graphics slot is a field of the layout, so the InfiniteReality pipe fits where the test widget is now. Ip27Board loses its NAME constant, since one board type serves two products; the product's name comes from the profile. Both products are reached through DesksideMachine, and the two test modules become one that runs the shared cases against both layouts. No behaviour changes: the tests that read the flash words, slot identities and widget map for one and two nodes pass as they were. FlashValues::ORIGIN200_R10000_180_1M was the base for the Origin 2000 and the Onyx2 as well as the Origin 200, each overriding the machine type and the processor module's words. The values are the stock PROM image's, not one product's, so the constant is STOCK_R10000_180. Asking Deskside for the layout of a profile that is not a deskside is an error on the config path, not a panic.
The Origin 2000 module reported its power, reset and NMI lines as ModuleEvent. The machine's ModuleController wanted the Origin 200 controller's ElscEvent, so the deskside translated one into the other and dropped the reset line's falling edge on the way. Both controllers now emit one type, renamed ModuleOutput because two controllers send it, and ModuleManagement hands the module's outputs on as they are. controller_output remains the one place that maps an output to a MachineEvent. The module still holds its reset line high for the 100 ms of a pulse, but reports only the rising edge, as the translation already did; no machine reads the fall. The reset test checks that the fall is silent.
The Onyx2's graphics slot held a TestWidget in production code, a second "log and fail" bus device beside ip27_machine's Unmodelled, with its own log text. Unmodelled takes an optional widget identification word that it answers at offset 4, and the slot uses that; TestWidget goes. The log line still says the access was to the graphics slot, which holds a test widget.
pci_devices took nine positional arguments, three of which (the scsi name, the product name and the Bridge) every caller derived from the configuration or built just before. PciPopulation holds the five things a board actually chooses (the Bridge, the IOC3 and Ethernet it fits, the module controller and the SuperIO pin), and the two names come from the profile inside pci_devices. BoardIo's test-only bridge field stays. The tests attach a PCI function to the Bridge's bus and check that dropping a machine frees it, and the Hub's widget table holds the Bridge as a dyn BusDevice behind a Crossbow port, which gives them neither.
Three places walked the banks the same way: skip the empty ones, number the RAM of the fitted ones, split a bank into its runs and track where each run starts in its RAM. NodeMemory::new built the decode table that way, Home built its lookup table that way, and NodeZero::apply repeated the walk every time a node's number changed. A change to the stride rule needed three edits. layout() makes the walk once, as a list of Runs stored in NodeMemory and shared by Home, NodeZero and memory_map. The memory map's sixteen-entry name table, which existed only to give each run a &'static str, becomes one name per bank.
The module notes said unfitted memory reads zero and drops writes, "logged under phys hole on". It was not: the default device had no log path, and phys hole on only toggles BlackHoleRegion, which reads all ones. A guest reading a bank that was never fitted left no trace. Unfitted memory now fails every access with a bus error and logs it under the hub category, which is the one the SN0 guides tell a user to turn on. The zero was an approximation nothing supported. The basis for the fault is a software expectation: the IRIX kernel's handler for an uncached bus error lists memory that does not exist as a case, and SGI's Embedded Support Partner event catalogue carries "Access to non-existent memory address". No capture from a physical machine shows it. The PROM's memory sizing does not need zero: it reads the directory back door, which answers its own fixed pattern for an absent bank, so it never reaches unfitted memory. Boots to an IRIX shell with the hub log on, with one 128 MiB bank, two, and a 256 MiB bank on the R12000 module (four 64 MiB runs with three holes between them), log no unfitted access. Only addresses outside the fitted runs reach the default device, so fitted memory pays nothing for the change. The test checks the contract at every width and no longer drives the process-wide log with retries.
The module controller's logs (the Origin 200's ELSC, the Origin 2000 module's controller, their configuration NVRAM and the I2C controller) went out under LogModule::Bedrock, which debug.md describes as the Fuel's devices on its Bedrock hub. They are SN0 parts, so a user turning on bedrock to watch a Fuel got controller lines, and a user trying to watch an Origin 200's controller had to know to turn on a Fuel category. They now log under a new module category, and debug.md lists it with hub, which the SN0 guides tell users to enable and the list left out. bedrock stays for the Bedrock devices. An unknown name in IRIS_DEBUG_LOG prints "unknown log module" at startup and is otherwise ignored, so debug_log = "bedrock" on an SN0 machine still starts and now enables nothing. A round-trip test pins the name table, the category list and the count together.
The machine woke every 10 ms for the life of the guest to bring the module controller up to date, even when the controller had nothing pending. The controller keeps its own time, and a guest's bus access already replays what is overdue, so the visit only matters when the bus is idle while a countdown or a pulse ends: an operator typing at the AUX port, a power-down running out. Elsc and Origin2000Module now say when they next have an output to give or a committed action to run (next_due), and tell the machine when an access may have moved that (set_wake). ControllerWatch holds one one-shot timer, armed for that time, re-armed after each visit and each wake, and removed when the machine stops, as the Hub's real-time comparator does. The grant ring's ticks and a message's expiry change only what a bus access sees, and the access replays them first, so they are not due times; an idle controller arms nothing. New tests drive the controllers on their explicit clock: an NMI wait and pulse, a power-down countdown and the module's reset wait and pulse report the right deadline, and an access wakes the machine only when the schedule moved. The machine test uses a scripted controller to check that a visit happens when due, a moved schedule replaces the timer, and a stopped machine holds none. A bus access can pass a deadline before the machine's visit. It then queues the output itself and clears the deadline, and with nothing due the machine would arm nothing: a power-off, reset or NMI would wait for some later access to move the schedule. The controller and the module therefore also report themselves due, at once, while an output they queued has not been taken, and the watch never arms a zero-length timer, so a controller that stays due cannot spin the timer thread. A test lets an access beat the visit and checks that the queued outputs are due.
Iris had three hand-written AMD command state machines: AmdCore (landed for the Fuel, no consumer upstream), the node flash's, and the BaseIO flash's. AmdCore now takes a Geometry: bus width, sector count and size, the unlock addresses and how many bits of them the part compares, the autoselect codes, whether it answers a CFI query, whether program and erase are refused, whether erase takes time, and where an unsupported cycle is logged. The default is still Fuel's 8 MiB x16 part, and its public names (Default, Effect, SECTOR_SIZE, PART_SIZE, the erase times) are unchanged, so a user of the engine compiles as it did. NodeFlash (1 MiB x8, unlock 0x5555 and 0x2aaa, codes 0x01 and 0xd5) and BaseioFlash (1 MiB x16, unlock matched on 11 bits, codes 0x01 and 0x22da, read-only) become owners of an AmdCore and keep only their storage: the node flash applies a program or a selected erase and writes it through to its file, the BaseIO flash serves its image. Their tests pass unchanged. The file format is the raw 1 MiB image as before. One behaviour follows from sharing the engine. The node flash treated 0xf0 as a reset even where it was the data cycle of a program, so a programmed byte of 0xf0 never reached the array; the engine, like the part, takes every bit of the data cycle, and a test pins it. The node flash still allows a new command sequence from autoselect without a reset, which the others do not, as a geometry flag. The part's data sheet agrees (AMD Am29F016D, publication 21444 rev E+2, page 16, "Reset Command" and "Byte Program Command Sequence"): a reset is accepted between the cycles of a sequence before programming begins, and the program address and data are the fourth bus cycle, so a data byte of 0xf0 is data. The data sheet checked is the Am29F016D's; the node PROM's part is an Am29F080 of the same AMD command family, which was not available to check. The shared engine also logs and cancels a sequence that gets a wrong address on an unlock cycle, where the node flash's own code silently went back to array mode.
The Hub is one ASIC with four register sections, but it read as five
devices: hub.rs and hub_{pi,md,io,ni}.rs were flat public modules, and
machines reached into the units with hub.io().connect_widget(..),
hub.pi().set_timer_manager(..), hub.md().set_onewire(..) and a raw
register offset to enable a processor. The units are plain parts of one
chip, as the mgras board's are.
They now live in src/dev/hub/{mod,pi,md,io,ni}.rs with their tests beside
them, the unit modules private, and drop the chip prefix inside the
module (hub::pi::Pi, not HubPi). The Hub gets the methods a machine
needs (connect_widget, connect_packets, set_timer_manager, set_onewire,
enable_cpu), and SlotSignals and the register offsets machines and their
tests name are re-exported from hub. Log names and the locks monitor's
names are unchanged. The commit is the move and the narrowed surface; no
behaviour changes.
The Hub carried a mechanism for a processor-rendezvous hold that nothing builds: HubObserver and Store, the observe() calls in the processor, I/O and network units, Hub::set_observer, and the observer tests. It also had Hub::attach_uart and a UART overlay over the memory block, which no machine fits; Hub::set_powered, which no machine calls (the module controller's power output is a machine event, not a Hub input); and HubConfig::nasid, a node number preprogrammed for a synthesized firmware that does not exist here. Only tests used any of it, and the audit expected another branch to; that branch does not. All of it goes, with the tests that exercised only it. Every register test stays. The network unit's enable switch, which two unit tests use to power a peer down, is now built for tests only.
The Hub's four units were joined to the Hub, and to each other, by back-references: the processor and network units each held the machine's hooks through a forwarding shell that existed to copy the cached-alias size into the Hub; the network unit told the Hub to rebuild a 256-entry claim map through a second trait object; and the I/O unit's block transfer engines reached memory and interrupts through a stored Weak<dyn NodeFabric> to the Hub. Machines also saw hooks for the processor enable and NMI stores that only logged. A unit's register write now returns what it changed, as the AMD flash engine returns its program effect, and the Hub acts on it once the unit's lock is released: the node number and cached alias reach the machine, a soft reset reaches the enabled slices, a restart reaches the machine, and a launched transfer runs with the Hub passed as its fabric. Writes through the node's I/O space take the same path, through a weak handle the space holds only to write. The enable and NMI stores log in the processor unit, which is where the log line belongs. The claim map goes. The network unit mirrors its node number and its neighbour's in two atomics when either changes, and the Hub's decode compares a node number with them, so a decode still takes no lock and the rebuild, its trait and the 256 stores per renumbering are gone. A vector request that renumbers the neighbour comes back to the launching Hub as an effect, which tells the neighbour's machine through the hooks the two share. The XIO link's reset closure, which no machine ever connected, goes too: a warm reset is logged, as it was. Unit tests drive the units directly and read the returned effects; a new Hub test checks that a write through the I/O space acts like one on the window.
Every store to PI_INT_MASK*, every device interrupt the Bridge sends (PI_INT_PEND_MOD) and every real-time pending or enable change published the processor's three lines under a reentrant lock, then the controller mutex, then one mutex per line: eight lock acquisitions for a change, whether or not any line moved. The controller behind them, CpuIrq, only set or cleared a bit of the processor's interrupt word, so neither mutex protected anything the dispatch lock did not. The Hub now writes the word itself (MipsCpu::interrupts_ptr passed to set_cpu_interrupts). Each input is a sequentially consistent atomic. One publisher at a time writes a processor's word. A caller stores its input and then publishes; a publisher that arrives while another is writing counts itself and returns, and the writer samples the inputs again before it leaves, so every change is sampled by a writer that began after it. The samples are taken in time order by one thread, so the word follows the registers in the order of those samples, and one pulse of an input is at most one rising edge in the word. A sample reads the masks, the pending banks and the comparator one after another, not as one snapshot, so a level can be one that no single instant held. A level can also outlast its input until the writer's next sample. Nobody waits and nothing is locked, so the dispatch lock, the controller mutex, the per-line mutexes and CpuIrq go. (HEART publishes under its register lock, which this does not copy.) The pending banks and masks were already atomics; their stores and the comparator and enable bits move from release and acquire to sequentially consistent. A first version wrote from a sample and re-checked the inputs afterwards. It converged, but between a stale write and its correction a processor could see a level the registers had dropped: in a probe, a stale high after a clear in 1 round of 3000, a second rising edge for one pulse and so a spurious interrupt. Serialising the writers removes that. A publish costs the input loads, one load of the word and two atomic updates of the Hub's own arrival count; it writes the processor's cache line only when a line changes. A booted Origin 200 under IRIX 6.5.30, from the PROM menu through a login and a minute of find, cat of the kernel and 40 process spawns, counted 23 stores to PI_INT_MASK*, 60211 interrupt-modify calls and 65051 publications: roughly 520000 lock acquisitions that no longer happen. IRIX does not write the mask on every priority change, so the cost was in the interrupt stream, not the mask. The instructions the CPU thread retired in user space over that minute (perf stat on the running process, the host at a load average above 190, release builds, one boot each, ordered base, branch, branch, base) were 423.4, 408.6, 380.2 and 371.7 billion: 397.5 against 394.4 billion on average, which is inside the 7 percent the same build varies by, since the work the guest does in a fixed wall-clock window depends on its timers. The change is not shown to be faster or slower; it removes the locks and their ordering rule. That measurement was taken with the first version of the publish; the arrival count was not measured. The word is a leaf, so a publication cannot reenter the Hub. A test runs the stale-sample case deterministically: an input falls between the writer's sample and its write, and the word shows one rising edge. A second counts rising edges while two threads keep publishing against pulses of an input, and a third races an input, a mask and an enable on three threads and checks the word against the registers. The word must outlive every thread that can publish to it. A machine stops its processors, devices and the Hub's timers first, clears the words in Drop, and only then frees the processors, since a cabled Hub can still reach this one's units; a publisher that loaded the pointer before the clear and ran after the free would write freed memory, which the stop rules out. A test holds a Hub past its machine to check that nothing publishes into the freed processor.
The Origin 200 guide still said the Origin 2000 was not built, the README's machine row listed none of the three SN0 machines, and the machine module's notes described a single-node, Origin-200-only machine. Several comments named the Origin 200 for things all three share: the processor and module tables, the bank count, and the cable between two node boards. HACKING.md and the BusDevice::write_narrow_from documentation said that only Bedrock's ECC view reads the source register's other lanes. No device in this tree overrides it yet, and the call is a dyn call where the uncached byte and halfword stores used to call write8 and write16; both now say so. A clanker note records two rules a reader would otherwise have to rediscover: a renumbered node sends node zero's cached window beyond its alias to the Hub that holds zero, which costs it the ppmem fast path, and a Bridge behind a Crossbow picks its host by the widget the guest programs, gated by that port's link. CHANGELOG gets one SN0 entry. The cost of the second node is measured and recorded in the Onyx2 and Origin 2000 guides: under IRIX 6.5.30 on two nodes, a CPU-bound awk loop took 28.1 s on average on the first node's processor and 61.8 s on the second's (five runs each, alternating in one boot), about 2.2 times as long. Nothing changes in the code for it.
The Hub still asked the machine four things through a hooks trait: the node number, the cached alias, a processor soft reset and a restart. A trait that a board implements is the wrong direction for a chip, and each call hid ordering: the effect carried the new value, so two writers that raced could leave the machine's routing at the older one. The node number and the cached-alias size are now published values the owning unit stores under its own lock, as the register changes. The Hub's address decode reads them from there with no lock, as before and through one pointer. The machine's node-zero routing is handed to the Hub once and re-reads both under its own lock when a store, a neighbour's vector write or a reset has changed one, so the last value written wins. A soft reset is a line, not a call: the Hub sets bit 63 of the processor's interrupt word, which is what the processor's reset signal did, without a lock and without the machine. A restart goes out on the machine's event channel as a hard reset, wired the way the Indy IOC's power events are. Enable and NMI stay stored and logged. The node number stays 16 bits because the register field is 9 bits wide. No per-access cost is added: the claim decode loads the number and the alias through a pointer it already followed.
src/ip35.rs held the bounded MMIO windows, their sparse map and map_device, but the only code that uses them upstream is the SN0 node bus, so IP27 imported its bus library from a module named for the Fuel's software platform. The file becomes src/window_map.rs, named for what is in it, and takes the physical address masks that ip27.rs and a Fuel bus both define (PA_MASK and CPU_PHYS_MASK), so there is one definition. A bus that wraps the four views differently stays with the machine that has it. The move is a rename; a test now pins what the window map did without one: a window answers exactly its bounds, the rest of the page is refused, and a later map over a page replaces the earlier.
On a two-node Onyx2 or Origin 2000 the PROM numbers one node 1. That
node's software addresses its own memory by its number (bit 32 up), and
node zero's memory beyond the cached alias belongs to the other node.
The processor's cache and the JIT reach RAM through the node's ppmem
window, but the window covered only node-relative offsets below 4 GiB,
and NodeZero cleared it entirely once the node held a nonzero number.
Every cached access of that node's processor then went through the
decode table, the Hub's node view and its bank lookup, and every
instruction fetch missed the decode cache's window pointer and re-read
its word over that path. A per-thread count in a throwaway build showed
it: during `runon 1 awk 'BEGIN{for(i=0;i<1000000;i++)s+=i}'` on the
Onyx2, all 912 million instruction fetches and 458 million data
accesses of that processor took the bus path, 86 percent of the
fetches to its own memory at its number and the rest to node zero's
memory. During runon 0 the other processor fetched 665 million of its
804 million instructions from node 1's memory the same way.
Windows now lays every node's window from the numbers and aliases the
Hubs publish, as the decode table routes them: node zero's window from
the node's own banks below the alias and the zero holder's above it,
and each node's memory at its number, the other node's included,
sharing its storage and generation counters. A two-node board gets an
8 GiB window, so numbers 0 and 1 fit. Windows lays only the difference:
a page that stays mapped is replaced in one mapping call, and a range
that goes leaves the region bitmap before it is scrubbed, so a
renumbering never clears a window the other processor is executing from.
The access path is unchanged.
A ppmem space takes no lock, because its processor's CACHE instructions
change the space's views and the space is single-writer by contract. A
store to a Hub's number or alias can come from the other node's
processor (the neighbour's vector write), so laying every window from
the storing thread would remap the other node's space under it. The
store now publishes the numbers, which the Routing atomics already did,
and marks every window stale. The storing processor lays its own node's
window at once if the store was on its node, and each processor lays its
own at its next batch boundary through the executor's batch hook, set
only on a board with several nodes. A window can trail the decode table
by one batch, about a thousand instructions, on the processors that did
not store. The PROM and the kernel renumber while booting, and a store
is not ordered against another processor's accesses without a
synchronisation the guest does. A mutex in PpMemSpace was the
alternative; it would have put a lock the hardware does not have in the
memory model. Two tests renumber on one thread while another settles
and holds and releases line pages. With every window laid from the
storing thread, as before, the first fails and the second aborts the
test process (SIGILL, three runs of three); with this arrangement both
pass.
With the fix the same count shows no cached access on the bus path on
either processor, from power-on to the end of the runs. On the Onyx2
at this branch's previous tip, the awk loop took 29.9 s on processor 0
and 57.9 s on processor 1 (means of five runs each, alternating in one
boot). With the fix it takes 27.9 s and 28.2 s. Boot to the login
prompt went from 293 s to 208 s. The two-node Origin 2000 runs the
loop in 35.2-36.5 s on processor 0 and 37.7-39.6 s on processor 1.
An 8 GiB window has 128 MiB regions in its bitmap, so on two nodes a
64 or 256 MiB bank, which fills no region whole, stays on the bus path
for both processors. A number above 1 is left to the bus path too.
Shipped machines: the Indy, Indigo2, Octane and O2 are not rewired, but
their ppmem changes. A mapping now keeps a reference to its backing
memory instead of a bank number, and recording a mapping trims one it
only partly overlaps, where it used to drop only mappings it covered
whole. That runs when firmware remaps memory, not on a load, store or
fetch, whose path is unchanged.
The Origin 2000 guide said a two-node machine stops in IRIX with "Processor #1 did not start", the second node's processor taking a TLB exception in the PROM. It no longer does. At this branch's tip, with and without the node window change, a two-node Origin 2000 (180 MHz R10000, 128 MB per node, stock IP27 PROM 6.156 and I/O PROM) boots IRIX 6.5.30 from the BaseIO's SCSI disk to a root login on the serial console, and hinv lists two IP27 processors. runon 0 and runon 1 both complete.
The two-node SN0 window was 8 GiB, so its one-word region bitmap counted 128 MiB regions. The bitmap only names a region that is RAM end to end, and a 64 MiB bank fills half of one, as does each 64 MiB run of a 256 MiB bank. On a two-node Origin 2000 or Onyx2 with those banks every cached access of both processors took the bus path again, a regression for the first node, which had used 64 MiB regions before. A node number above 1 fell outside the window altogether. The bitmap test runs on every load, store and fetch, so it stays as it is. A space now keeps a fine table beside it, one bit per 64 MiB region of the whole window, and the shadow cache consults it on the bitmap's miss branch before taking the bus. A space's bitmap region size can be set apart from its window size. A cabled SN0 board gets a window spanning the node numbers up to the next power of two above its node count, 8 GiB for two nodes and 16 GiB for three or four, with 64 bitmap regions of 128 MiB over numbers 0 and 1. One node keeps its 4 GiB window and 64 MiB regions. The window is reserved address space that commits nothing (mmap PROT_NONE and MAP_NORESERVE on Unix, a VirtualAlloc2 placeholder on Windows, read from the code and run only on Linux), and sizing it by the node count rather than for all 256 node numbers keeps the reservation at 8 GiB per node on a two-node board. A node a guest gives a number past the window still works: the bus path answers every number, at the speed it had before the window. The decode cache keeps window pointers for pages either test finds. It used to key them on the bitmap and start over when it moved, which misses a change only the fine table sees, so it now keys them on a value the space bumps whenever either changes (PpMemSpace::add_key_sink). Every machine that wires a shadow cache to a window registers it. With a throwaway per-thread counter, a two-node Onyx2 made no cached access on the bus path on either processor, from power-on through a runon 0 and a runon 1 awk loop, with two 64 MiB banks per node and with one 128 MiB bank. The loop ran at the same speed on both processors in both layouts (means 19.3 s and 19.8 s at 64 MiB, 19.4 s and 18.1 s at 128 MiB, five runs each, alternating in one boot; the spread within a boot is 17 to 24 s, so differences under about 20 percent are noise). Shipped machines: this changes the Indy and Indigo2 (machine.rs), the Octane (ip30_machine.rs) and the O2 (ip32_machine.rs) as well, since they share the shadow cache and the decode cache. Each registers its CPU's key sink with add_key_sink(view, ptr). On the bitmap's miss branch the shadow cache makes one more compare, against the fine table's length, which is zero on these machines because only SN0 sets a table, so the branch falls through to the bus as before. The decode cache's hit compares the window key where it compared the bitmap, one compare as before. The bitmap test itself and the window pointer are untouched. One node, `iris-bench --deterministic --cpu r10000`, host instructions:u, A B B A, where A is upstream main at the branch point and B is this branch's tip: 343.10e9, 339.47e9, 339.14e9, 342.87e9, with identical guest work (621.45 million instructions) on every run. The tip runs about 1.1 percent fewer host instructions. That compares the whole branch, so it bounds the combined effect of its commits, not this one alone, and it was taken before processors laid their own windows, code only a board with several nodes runs. On a one-node Origin 200 under IRIX 6.5.30, the processor thread's user instructions over one awk loop, base and branch alternating over six boots (eleven runs), were 243e9 to 279e9 for the base and 239e9 to 292e9 for the branch, means 264e9 and 251e9. That count is not fixed work, since the guest's timers run on the host clock, so it shows no regression beyond its own spread and nothing finer.
A 512 MiB bank 0 spans the whole 512 MiB stride, boot flash's reset
address included. The node memory mapped the flash over the bank in the
cached decode and listed both in the memory map, so MemoryMap::new
panicked ("bank 0 overlaps boot flash") for any machine built with one,
such as a two-node deskside with 512 MiB in bank 0.
Cached references now reach the bank, as the PROM's memory test expects
when it walks that address as RAM, and the reset fetch, which is
uncached, reaches the flash through the Hub-special space, where it
already is. With a smaller bank 0 the flash stays in the cached decode
as before. The QEMU sgi-origami node model makes the same split. That a
real machine does is inferred from the PROM's memory test, not measured.
A test builds every size bank 0 can be, and checks that a 512 MiB one
keeps its RAM at the reset address and every smaller one the flash.
jamesbraid
force-pushed
the
ip27-fixes
branch
from
November 10, 2026 17:00
ef7758d to
434abe4
Compare
jamesbraid
changed the title
jamesbraid
force-pushed
the
ip27-fixes
branch
from
November 10, 2026 19:17
434abe4 to
a3a9bbd
Compare
jamesbraid
force-pushed
the
ip27-fixes
branch
from
November 10, 2026 23:11
a3a9bbd to
2d56631
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Builds on #507 (the IOC3 refactor); the first 14 commits here are that PR's.
The IP27 machines from #455 boot, but the code carried shims, a lock-heavy Hub, two config sections for three machines and some bugs found bringing up graphics on the Onyx2. This cleans that up and fixes them.
Fixes
io_promfits the I/O PROM on the BaseIO's flash, without which the node PROM runs a copy with no graphics console.Cleanup
[origin200],[origin2000]and a new[onyx2]read into one struct. An Onyx2 file that still says[origin2000]gets an error naming[onyx2], so existing Onyx2 configs need that table renamed.src/dev/hub/) with private units. Units return what they changed instead of calling hooks; restarts go on the machine's event channel; soft reset is a line on the processor's interrupt word. Interrupt publishing takes no lock (it took about eight per mask write).xtalk::Upstreamreplaces, and the Origin 2000 Bridge that ignored the Crossbow link.src/ip35.rs's window map moves tosrc/window_map.rs.Worth a look: the lock-free Hub publisher (
hub/pi.rs) and the node windows (ip27.rs,ppmem).Tested with
cargo test --workspace(also the IP27 tests underjitv2andlightning,jitv2,idle-pause), every commit building. IRIX 6.5.30 boots to a shell on an Origin 200, a two-node Origin 2000, a two-node Onyx2 (with the MediaIO's serial ports and audio inhinv), an R12000 with a 256 MiB bank and a 512 MiB first bank; an Octane still boots. Not built on macOS or Windows here.