diff --git a/CHANGELOG.md b/CHANGELOG.md index 76ea524a1..b6b51e7ed 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -13,6 +13,13 @@ hashes are given where a change is easiest to understand by reading the commit. ## November 2026 +### R10000-family cache lines as memory + +- A valid primary data or secondary cache line now holds its page in that CPU's own copy of ppmem's window, so a PROM that keeps its stack in cache lines reads and writes the lines with cached accesses while the uncached path, DMA and the other CPUs see memory. Index Store Data writes through to such a line. `l2 probe 0` lists the pages held. Nothing changes on a load, store or fetch. A valid R5000 primary data line does the same, for the O2's POST stack. +- Primary instruction lines hold their pages too, each word written there as the instruction it predecodes, for the eleven predecode classes the model knows. Index Load Data (I) where a held secondary line would refill the set returns that line's words, predecoded, until Index Store Data (I) writes the set. Only the Fuel's and Tezro's PROMs use them. +- jitv2 does not compile a page any CPU's cache holds lines in: that CPU runs it from its own view in the interpreter, and every other page stays compiled. +- The O2 keeps POST's stack in those lines. CRIME's side page for the uncached path and the tag-store hook are gone, and an O2 snapshot an older iris took during POST is refused. + ### R10000 family fixes, R12000 and R14000 - The R10000 family has 44-bit virtual and 40-bit physical addresses, FrameMask and a 28-bit EntryLo PFN, picks the TLB refill vector by address space, invalidates overlapped TLB writes (Status.TS), and completes uncached stores nothing answers. Partial LWR keeps the upper word, and the cache diagnostics (Index Store Data, secondary tag mask, Config size) work. diff --git a/clanker/INDEX.md b/clanker/INDEX.md index b64133f49..5d8921f6c 100644 --- a/clanker/INDEX.md +++ b/clanker/INDEX.md @@ -24,6 +24,7 @@ CPU architecture: CP0, TLB and its fast paths, caches, FPU, LL/SC, CPU models. B | Note | Kind | Title | |---|---|---| | [cached-flash-is-a-cache-with-two-cpus.md](cpu/cached-flash-is-a-cache-with-two-cpus.md) | rule | With two CPUs, cached flash reads must hold their lines: a fetch through to a chip in status mode executes `0x80808080` | +| [cache-lines-live-in-the-cpus-view.md](cpu/cache-lines-live-in-the-cpus-view.md) | design | A valid R10000-family cache line, or R5000 primary data line, makes its page private in that CPU's own copy of ppmem's window; the uncached path and other masters keep memory. Limits and how to see what is held | | [cache-attributes-and-fetch-alignment.md](cpu/cache-attributes-and-fetch-alignment.md) | rule | Cache attributes and fetch alignment: three hardcodings, all wrong | | [cache-tag-index-and-check-bits.md](cpu/cache-tag-index-and-check-bits.md) | rule | Cache tags: index ops index by address; check bits are generated, not stored | | [cop0-requires-kernel-or-cu0.md](cpu/cop0-requires-kernel-or-cu0.md) | rule | CP0 instructions must be gated on Kernel-or-CU0 | @@ -126,7 +127,7 @@ Non-graphics devices: MC/MEMCFG and RAM banks, HPC3, VDMA, EEPROMs and NVRAM, PS | [aic7880-patchsim.py](devices/aic7880-patchsim.py) | howto | Reproduces the Linux aic7xxx firmware patch selection and critical sections for a 7880 | | [o2-no-ecc-alias-is-memory.md](devices/o2-no-ecc-alias-is-memory.md) | rule | O2: physical 0x00_8000_0000 is CRIME's no-ECC memory alias, folded onto RAM by the CPU (netstat/Netscape hang, /dev/mem all ones) | | [o2-cache-handover-full-to-shadow.md](devices/o2-cache-handover-full-to-shadow.md) | history | O2 cache as it was until 2026-10-30: the full model through POST, the shadow model after, switched by PC range and a tag-store trigger | -| [o2-prom-stack-is-a-cached-page-at-phys-0.md](devices/o2-prom-stack-is-a-cached-page-at-phys-0.md) | design | O2 PROM stack: one tag-installed page at physical 0 that the memory test overwrites through KSEG1; a 4 KB page in the bus decode answers for it uncached, so the shadow cache runs from reset | +| [o2-prom-stack-is-a-cached-page-at-phys-0.md](devices/o2-prom-stack-is-a-cached-page-at-phys-0.md) | design | O2 PROM stack: one tag-installed page at physical 0 that the memory test overwrites through KSEG1. Retired: the CPU's cached view holds it now | | [memcfg-limit-is-not-the-bank-size.md](devices/memcfg-limit-is-not-the-bank-size.md) | rule | MEMCFG: `limit` is the slot, `addr_mask+1` is the mirror period — they differ | | [nvram-persistence.md](devices/nvram-persistence.md) | design | Battery-backed state on Stop | | [ps2-aux-mux-probe-and-status-bit.md](devices/ps2-aux-mux-probe-and-status-bit.md) | rule | PS/2 mouse not detected under Linux 3.2 guests (and keyboard sometimes flaky too) | diff --git a/clanker/cpu/cache-lines-live-in-the-cpus-view.md b/clanker/cpu/cache-lines-live-in-the-cpus-view.md new file mode 100644 index 000000000..d4a157db0 --- /dev/null +++ b/clanker/cpu/cache-lines-live-in-the-cpus-view.md @@ -0,0 +1,104 @@ +# Cache lines live in the CPU's own view of memory + +The R10000-family PROMs use the caches as memory before memory works. They make +lines valid with Index Store Tag and Index Store Data and then load, store and +fetch through them, while uncached accesses, which bypass the caches (NEC +VR10000 user's manual p. 75), and every other master see memory. What the shipped PROMs do on iris: + +| Machine | Lines | Memory behind them | +|---|---|---| +| O2 | secondary and primary data lines for page 0 and 0x1000 hold POST's stack for about 50-65 ms, while POST tests page 0 uncached. On the R5000, primary data lines for page 0 (TagLo 0xc0) | yes | +| Octane | a tag and data RAM test on secondary set 0, read back only with Index Load Tag/Data | no: the patterns name no memory | +| Fuel | primary data lines for 0x100000-0x107fff hold the stack; primary instruction lines hold PROM code | no, until memory is configured mid-phase | +| Tezro | the same as the Fuel, on every processor at once, with different stack contents at the same addresses | the same | + +Every line ends by Index Store Tag. None is written back, and none is read back +with Index Load Data while it holds a page. + +## How + +- Each CPU reads memory through a window of its own (`PpMemSpace::add_cached_view`). + Every mapping laid in the shared window is laid in each view too, and the view's + region bitmap goes to that CPU's inline fields. The shared window, which + `Physical`'s direct paths, CRIME's rendering engine and HEART read, never holds a + line. +- When Index Store Tag makes a line valid, its page becomes private memory in that + view, a copy of memory, and the line's data array is written into it + (`ShadowCache::line_enters`). An invalid or different tag ends the line, and the + page goes with its last line. The shadow's tags stay the only line state, with a + held bit beside each. Index Store Data into a held line writes through. +- The R5000 does the same for a valid primary data line, whose tag has the + R10000's PTag and PState fields (NEC VR5000 user's manual U11761EJ6V0UM p. 133). + It has no Index Store Data, so the page starts as memory. Its instruction lines + hold nothing, since one could hold only what memory holds. +- A page with no memory behind it holds nothing: the line stays in the arrays, as + a tag test's patterns do. The region bitmap is all a CPU tests, so a held page's + whole region is the view's while it stands, reading zeros wherever nothing is + mapped. The O2's POST holds lines for page 0 before CRIME maps the rest of that + region. +- A remap saves and restores held pages, so memory configuration mid-phase keeps + the stack. +- Loads, stores, fetches, the decode cache and compiled code are untouched: they + read the view through the base pointer and bitmap they always had. The decode + cache compares every word it serves, so it sees a page change hands. + +## Limits + +Each is an approximation, stated where it is made. + +- Page, not line: the rest of a held page shows memory as it was when the page was + taken, uncached writes to it meanwhile are not seen by cached reads, cached stores + to it are dropped at release, and a line that ends leaves its bytes until the + page's last line goes. The page is the host granule: 4 KB on Linux and x86 macOS, + 16 KB on Apple silicon, 64 KB on Windows. +- Lines end only by a tag change the shadow already models: Index Store Tag, and + Index Writeback Invalidate (D) zeroing the tag from the R12000 on. Nothing is + written back. The hit operations and the other index invalidates leave a held + line as they leave every shadow tag, untouched. +- Index Load Data reads the data array, not the page, so it does not see cached + stores. +- A region holding a page reads zeros in its unmapped gaps for that CPU's cached + accesses, where the bus would answer. +- A secondary doubleword puts TagLo at the lower address. The manuals say only + that TagHi is the high word and TagLo the low word (MIPS R10000 UM p. 201, NEC + VR10000 UM p. 289); which address each pairs with is inferred. PROM 6.210 (Fuel, + Tezro) stages code with the word at A in TagLo and the word at A + 4 in TagHi, + 990 times in a Tezro boot, and its staged code equals its image only this way. + The IP35 boot-flash PROM's staging pairs the words the same way: simulated with + the other order, 378 of its 562 staged fetches break, every trampoline fetch + among them. With the other order the Tezro's PROM ran its long cache test + uncached, taking twice as long. +- No replacement, parity or ECC, and no coherence with other CPUs or DMA. +- A snapshot taken while lines hold pages is refused at restore. + +## Compiled code + +A page some CPU holds is not compiled: compiles read memory, and compiled code is +shared. ppmem puts `GEN_HELD` on the page's generation counter at the first hold +and takes it off at the last release, moving it with the page on a remap +(`PpMemSpace::mark`). The compile that finds it publishes nothing and denies every +offset until the release. ppmem's counters are the bus's for memory; a machine whose bus +maps a device over memory, as the IP35 boards map their reset PROM, must mark that +device's counter as well. + +## Primary instruction lines + +Only the Fuel's and the Tezro's PROMs use these. A valid primary instruction line +writes each 36-bit predecoded word (p. 288) into its page as the instruction it +predecodes, through an eleven-class table inferred from PROM 6.210's words +(`l1i_decode`). A word of another class leaves memory's word, where hardware would +run it. A page held only for code also answers cached loads with code. + +The model never refills on a fetch. The Tezro's PROM runs a staged secondary line +and reads the primary words back with Index Load Data (I) to stage them itself, so +Index Load Data (I) in a set a held secondary line maps to answers with that line's +words predecoded (`refilled_word`), until Index Store Data (I) writes the set. When +two held lines map one set, the later wins. An instruction no class holds comes back +in the model's own form (bits 35:32 set, which takes classes 0x78-0x7f), which +hardware would not produce. Index Load Tag (I) reads the stored tag. + +## Seeing it + +`l2 probe 0` in the monitor prints how many lines hold pages and which. Every +machine should show none at the IRIX login. `log l2c mask op` logs each page held +and released. diff --git a/clanker/devices/o2-prom-stack-is-a-cached-page-at-phys-0.md b/clanker/devices/o2-prom-stack-is-a-cached-page-at-phys-0.md index 5356cce38..39f6e935b 100644 --- a/clanker/devices/o2-prom-stack-is-a-cached-page-at-phys-0.md +++ b/clanker/devices/o2-prom-stack-is-a-cached-page-at-phys-0.md @@ -1,10 +1,11 @@ # O2 PROM stack is one cached page at physical 0, and RAM behind it is a different thing -Status: landed 2026-10-30. Both O2 CPUs run the shadow cache from reset; the handover -([o2-cache-handover-full-to-shadow.md](o2-cache-handover-full-to-shadow.md)) is gone. Code: -`O2Memory::stack_tag_store` in `src/dev/crime_mem.rs`, the tag-store hook in -`src/cpu/mips_cache_shadow.rs`. Sections below are the measurements that chose the design; the -prototype results used earlier binaries. +Status: retired 2026-11-08. The stack now lives in the CPU's own cached view of ppmem's window: +a valid primary data or secondary line holds its page there (`ShadowCache::line_enters` in +`src/cpu/mips_cache_shadow.rs`, `PpMemSpace::add_cached_view`), and the uncached path, the decode +table and the rendering engine keep seeing memory. `O2Memory::stack_tag_store`, its side page and +the tag-store hook are gone. The sections below are the measurements that chose the old design; +what they say about the PROM still holds. ## Where the stack is diff --git a/src/cpu/jitv2/comp.rs b/src/cpu/jitv2/comp.rs index d5d009ec8..595750f31 100644 --- a/src/cpu/jitv2/comp.rs +++ b/src/cpu/jitv2/comp.rs @@ -116,6 +116,8 @@ pub(crate) enum Snapshot { Taken(u64), /// A word could not be read (not RAM right now). Unreadable, + /// A CPU's cache holds lines in the page, under this generation. + Held(u64), /// The generation moved under every attempt. Busy, } @@ -129,6 +131,10 @@ pub(crate) fn snapshot_page(page: &PhysicalCodePage, bus: &dyn BusDevice, phys_b const MAX_SEQLOCK_RETRIES: u32 = 8; for _ in 0..MAX_SEQLOCK_RETRIES { let gen0 = page.current_gen(); + // A CPU's cache holds lines in the page (`GEN_HELD`). + if gen0 >= crate::cpu::jitv2::jitv2::GEN_HELD { + return Snapshot::Held(gen0); + } // A seqlock reader: the words are read after `gen0` and before // `gen1` (acquire), and every writer bumps the generation after its // bytes, with release. Then a generation that held means these are @@ -199,11 +205,27 @@ fn prepare_multi_entry_compile( // §13.3 step 2: seqlock-snapshot the page bytes. let mut words = [0u32; ENTRIES_PER_PAGE]; let gen_snap = match snapshot_page(page, &**bus, phys_base, &mut words) { - Snapshot::Taken(gen) => gen, + Snapshot::Taken(gen) => { + // The lines are gone: lift the held publish's denials (below). + if page.entry_gen() >= crate::cpu::jitv2::jitv2::GEN_HELD { + page.allow_all(); + } + gen + } Snapshot::Unreadable => { page.mark_prepare_bounced(); return PrepareOutcome::Done(false); // page not readable right now; retry later } + Snapshot::Held(gen) => { + // Publish no code for this generation and deny every offset, so + // arrivals interpret the page without asking again until the + // release moves the generation (`step_jit`'s denylist gate). + if page.publish(&[0; crate::cpu::jitv2::BITMAP_WORDS], std::ptr::null(), gen, 0, 0, req.compiled_for_fr1) { + page.denylist_all(); + } + page.mark_prepare_bounced(); + return PrepareOutcome::Done(false); + } Snapshot::Busy => { // A sufficiently hot self-modifying/shared-DMA page can bounce here // EVERY single attempt (gen keeps changing faster than @@ -976,6 +998,26 @@ mod tests { assert!(!page.is_denylisted(4)); } + /// A page whose counter carries `GEN_HELD` compiles nothing and denies + /// every offset, and compiles once the mark comes off. + #[test] + fn a_held_page_compiles_once_its_lines_go() { + let _stable = crate::cpu::jitv2::opcode_support::table_stable(); + let bus: Arc = Arc::new(AddiuDevice); + let counter = AtomicU64::new(crate::cpu::jitv2::jitv2::GEN_HELD); + let mut page = PhysicalCodePage::new(0, &counter as *const AtomicU64); + let mut compile = |page: &mut PhysicalCodePage| { + page.mark_requested(4); + let req = CompileRequest { page: page as *mut PhysicalCodePage, compiled_for_fr1: true }; + handle_request_for_test(&req, &bus, &mut Analyzer::new(), &mut Codegen::new()); + }; + compile(&mut page); + assert!(!page.is_runnable(4) && page.is_denylisted(4), "held: interpreted, not asked again"); + counter.fetch_add(1u64.wrapping_sub(crate::cpu::jitv2::jitv2::GEN_HELD), std::sync::atomic::Ordering::Release); + compile(&mut page); + assert!(page.is_runnable(4) && !page.is_denylisted(4), "released: compiled"); + } + #[test] fn a_second_incremental_compile_does_not_drop_the_first_entry() { // §13's one-function-per-page model fully replaces `func` on every diff --git a/src/cpu/jitv2/jitv2.rs b/src/cpu/jitv2/jitv2.rs index 6442262fa..01455afab 100644 --- a/src/cpu/jitv2/jitv2.rs +++ b/src/cpu/jitv2/jitv2.rs @@ -228,6 +228,13 @@ pub fn push_hot_request(hot: &crossbeam_queue::ArrayQueue, req: /// for this constant. pub const PAGE_SIZE: u32 = 4096; +/// Carried by a page's generation counter while some CPU's cache holds +/// lines there (`PpMemSpace::hold_line_page`): that CPU runs the page from +/// its own view, so it is not compiled (`comp::snapshot_page`). It comes off +/// with an ordinary bump, below the held page's last publish, so `publish` +/// takes any change of generation as new bytes. +pub const GEN_HELD: u64 = 1 << 48; + /// Number of possible entry offsets per page: one per 4-byte-aligned word /// (MIPS instructions are always word-aligned), i.e. `PAGE_SIZE / 4` (§2.4: /// `entry_bits` 16×u64 = 1024 bits, `entry_table` 1024 entries). @@ -1620,6 +1627,11 @@ impl PhysicalCodePage { for word in self.denied.iter() { word.store(0, Ordering::Relaxed); } } + /// Undo [`Self::denylist_all`]. + pub fn allow_all(&self) { + for word in self.denied.iter() { word.store(u64::MAX, Ordering::Relaxed); } + } + /// Clear `offset_word`'s `compiled` bit only — NOT `denied`. Two /// distinct callers: /// @@ -1854,8 +1866,9 @@ impl PhysicalCodePage { /// a reader that observes a `compiled` bit set (Acquire-paired via /// `is_runnable`'s `entry_gen` load) never observes a stale/null /// `func` for it. `entry_gen` is written unconditionally on every actual - /// publish, but only **numerically advances** when `snap_gen > - /// entry_gen` (a real page-bump-triggered recompile, §13.6/§13.3's + /// publish, but only **changes** when `snap_gen != entry_gen` (a real + /// page-bump-triggered recompile, or the first after `GEN_HELD` comes + /// off, §13.6/§13.3's /// discussion) — a pure entry-coverage publish against unchanged bytes /// (`snap_gen == entry_gen` already) rewrites the same value. Readers /// never need to distinguish the two: `entry_gen == current_gen()` is @@ -1909,7 +1922,8 @@ impl PhysicalCodePage { } let prev_entry_gen = self.entry_gen.load(Ordering::Relaxed); - let is_real_invalidation = snap_gen > prev_entry_gen; + // Not `>`: a page's generation drops when `GEN_HELD` comes off. + let is_real_invalidation = snap_gen != prev_entry_gen; // Code for the other FR mode replaces the old function outright, so // the old function's entries must go with it: its dispatch switch is // gone, and the new one only has cases for `new_entries`. diff --git a/src/cpu/mips_cache_shadow.rs b/src/cpu/mips_cache_shadow.rs index 7f056a863..c49045c89 100644 --- a/src/cpu/mips_cache_shadow.rs +++ b/src/cpu/mips_cache_shadow.rs @@ -30,6 +30,18 @@ //! The shadow is deliberately not consulted by `read`/`write`/`fetch`. If it //! ever needs to be, this type is the wrong shape and should say so loudly //! rather than growing a slow path. +//! +//! The R10000-family PROMs use the caches as memory before memory works: they +//! make lines valid with Index Store Tag and Index Store Data and keep a stack +//! or run code in them, while the uncached path sees memory (NEC VR10000 +//! user's manual p. 75). Such a line lives in ppmem's window rather than +//! here: the window this cache reads is the CPU's own cached view +//! (`PpMemSpace::add_cached_view`), and a valid line makes its page private +//! memory in that view, holding the line's bytes +//! ([`ShadowCache::line_enters`]). Loads, stores and fetches read the view as +//! they always did. The CACHE instruction is the only thing that moves a page +//! in or out, and the tags stay the only record of which lines are valid. +//! `clanker/cpu/cache-lines-live-in-the-cpus-view.md` has the limits. use std::cell::UnsafeCell; use std::sync::Arc; @@ -38,6 +50,7 @@ use crate::cpu::mips_cache_v2::{ cache_op_name, model, r4k_load_taglo, r4k_store_taglo, CpuModel, FetchInstrResult, MipsCache, C_ILT, C_IST, C_IWBINV, C_R10K_CBARRIER, C_R10K_ILD, C_R10K_ISD, CACH_PD, CACH_PI, CACH_SD, CACH_SI, }; +use crate::ppmem::PpMemSpace; use crate::traits::{BusDevice, BusRead64, BUS_OK}; use crate::devlog::{LogModule, CACHE_LOG_HIT, CACHE_LOG_OP, devlog_is_active, devlog_mask}; @@ -63,11 +76,15 @@ struct Shadow { /// — see `MRU_BIT`. Hardware state rather than tag storage, which is why /// it is kept here and re-applied on read instead of living in `tags`. mru: Box<[bool]>, + /// The line holds its page in the CPU's cached view + /// (`ShadowCache::line_enters`). + held: Box<[bool]>, } impl Shadow { fn new(lines: usize, data_words: usize) -> Self { Self { + held: vec![false; lines.max(1)].into_boxed_slice(), tags: vec![0u64; lines.max(1)].into_boxed_slice(), data: vec![0u64; data_words].into_boxed_slice(), ecc: vec![0u32; data_words].into_boxed_slice(), @@ -87,6 +104,71 @@ impl Shadow { /// nothing, because the way that is read is never the way that was written. const MRU_BIT: u64 = 1 << 63; +/// R10000-family tag states (NEC U10278EJ3V0UM pp. 285-287): the primary data +/// state in TagLo[7:6] and the secondary state in TagLo[11:10], 0 Invalid. +const PD_STATE_SHIFT: u32 = 6; +const SD_STATE_SHIFT: u32 = 10; + +/// The primary instruction state, TagLo[6] (NEC U10278EJ3V0UM p. 285). +const PI_VALID: u64 = 1 << 6; + +/// A primary instruction word is the instruction predecoded, 36 bits (NEC +/// U10278EJ3V0UM p. 288). Neither manual gives the layout. This one is +/// inferred from the words PROM 6.210 stores, `p` the word and `f` the +/// instruction: p[25:20] = f[5:0], p[19:15] = f[10:6], p[14:10] = f[25:21], +/// p[28:26] = f[28:26], p[35:29] a class that also gives f[31:29], and p[4:0] +/// and p[9:5] the rt and rd fields in the order the class gives. Only the +/// sampled classes decode, and every row is inferred. `addiu $4,$0,2` +/// (0x24040002) is 0x3_4420_0004. +/// +/// Rows: class, f[31:29], whether p[4:0] is rt, the f[28:26] values as a bit +/// mask, and the function fields of the SPECIAL members. +const L1I_CLASSES: &[(u8, u32, bool, u8, &[u8])] = &[ + (0x00, 0, false, 1 << 0, &[0x00]), // nop + (0x08, 0, false, 1 << 0, &[0x38, 0x3c]), // dsll, dsll32 + (0x18, 0, false, 1 << 0, &[0x25]), // or + (0x1a, 1, true, 0b1010_0011, &[]), // addi, addiu, ori, lui + (0x1e, 3, true, 1 << 1, &[]), // daddiu + (0x35, 5, false, 1 << 7, &[]), // cache + (0x36, 6, true, 1 << 7, &[]), // ld + (0x3f, 2, false, 1 << 0, &[]), // mtc0 + (0x40, 0, false, 1 << 3, &[]), // jal + (0x41, 0, false, 0b101, &[0x08]), // jr, j + (0x71, 0, false, 1 << 1, &[]), // REGIMM +]; + +/// The model's own form, bits 35:32 set over an instruction no class holds. +/// Hardware would never produce it. It takes classes 0x78-0x7f, so a real +/// word of those classes would be read as raw. +const L1I_RAW: u64 = 0xf << 32; + +/// The instruction a predecoded word encodes, or the class that is refused. +fn l1i_decode(p: u64) -> Result { + if p >> 32 == L1I_RAW >> 32 { + return Ok(p as u32); + } + let (class, op, funct) = ((p >> 29) as u8 & 0x7f, (p >> 26) as u32 & 7, (p >> 20) as u8 & 0x3f); + let &(_, hi3, rt_low, ops, functs) = L1I_CLASSES.iter().find(|c| c.0 == class).ok_or(class)?; + if ops >> op & 1 == 0 || (op == 0 && !functs.is_empty() && !functs.contains(&funct)) || (class == 0 && p != 0) { + return Err(class); + } + let (low, high) = ((p & 0x1f) as u32, (p >> 5) as u32 & 0x1f); + let (rt, rd) = if rt_low { (low, high) } else { (high, low) }; + Ok(hi3 << 29 | op << 26 | ((p >> 10) as u32 & 0x1f) << 21 | rt << 16 | rd << 11 | ((p >> 15) as u32 & 0x1f) << 6 | funct as u32) +} + +/// The predecoded word for `f`, or the raw form where no class holds it. +fn l1i_encode(f: u32) -> u64 { + let w = f as u64; + let (rt, rd) = (w >> 16 & 0x1f, w >> 11 & 0x1f); + L1I_CLASSES.iter().find_map(|&(class, _, rt_low, _, _)| { + let (low, high) = if rt_low { (rt, rd) } else { (rd, rt) }; + let p = (class as u64) << 29 | (w >> 26 & 7) << 26 | (w & 0x3f) << 20 | (w >> 6 & 0x1f) << 15 + | (w >> 21 & 0x1f) << 10 | high << 5 | low; + (l1i_decode(p) == Ok(f)).then_some(p) + }).unwrap_or(L1I_RAW | w) +} + /// R10000 only: ways per set, as CACHE index operations address them. Bit 0 /// of the index selects the way; the set starts above the line offset. The /// R4400 and R5000 index differently — see `tag_slot`. @@ -154,8 +236,14 @@ pub struct ShadowCache< /// Decoded instructions, by physical page. Never touches the shadow /// arrays above — see `decode_cache`. decode: UnsafeCell, - /// See [`MipsCache::set_pd_tag_store_hook`]. Consulted by `cache_op` only. - pd_tag_hook: UnsafeCell>>, + /// The cached view `tc_base` is, and its number in `PpMemSpace`: where + /// valid lines hold their pages. Null without one. + view_space: UnsafeCell<*const PpMemSpace>, + view: UnsafeCell, + /// For each primary instruction set, the 64-byte half of a held + /// secondary line that a refill there would load (`refilled_word`), or + /// `u64::MAX`. + refill: UnsafeCell>, } // Safety: the CPU thread is the only accessor, as for every other cache model @@ -212,7 +300,9 @@ impl< #[cfg(feature = "jitv2")] tc_gen: UnsafeCell::new(std::ptr::null_mut()), decode: UnsafeCell::new(crate::cpu::decode_cache::DecodeCache::new()), - pd_tag_hook: UnsafeCell::new(None), + view_space: UnsafeCell::new(std::ptr::null()), + view: UnsafeCell::new(0), + refill: UnsafeCell::new(vec![u64::MAX; Self::PI_SETS].into_boxed_slice()), } } @@ -340,6 +430,228 @@ impl< crate::dlog!(LogModule::L2c, "shadow: -> ISD slot={slot} data={phys_addr:#018x} ecc={:#x}", s.ecc[slot]); } + if s.held.get(self.tag_slot(sel, virt_addr)) == Some(&true) { + self.store_into_held_line(sel, virt_addr, phys_addr); + } + if sel == CACH_PI { + // The set now holds what the guest wrote, not a refill. + if let Some(r) = unsafe { &mut *self.refill.get() }.get_mut((virt_addr as usize / IC_LINE.max(1)) % Self::PI_SETS.max(1)) { + *r = u64::MAX; + } + } + } + + // ---- valid lines hold pages in the CPU's cached view ---- + + /// Primary instruction sets, as Index Load Data (I) addresses them. + const PI_SETS: usize = if IC_LINE == 0 { 0 } else { IC_SIZE / WAYS / IC_LINE }; + + fn view_space(&self) -> Option<&PpMemSpace> { + unsafe { (*self.view_space.get()).as_ref() } + } + + fn view_no(&self) -> usize { + unsafe { *self.view.get() } + } + + /// The physical address and length of the line `tag` names, if it is + /// valid. `va` is an index operand naming the line's slot: the primary + /// caches take PA[11:5] or PA[11:6] from it, the secondary PA[17:7] (NEC + /// U10278EJ3V0UM pp. 285-287; the tag holds the rest). The R5000's + /// primary data tag has the same PTag and PState fields (NEC + /// U11761EJ6V0UM p. 133). + fn line_of(&self, sel: u32, tag: u64, va: u64) -> Option<(u64, usize)> { + let pa_hi = ((tag >> 32) & 0xf) << 36; + match sel { + CACH_PD if DC_LINE != 0 && (tag >> PD_STATE_SHIFT) & 3 != 0 => { + Some((pa_hi | ((tag >> 8) & 0xff_ffff) << 12 | (va & 0xfff & !(DC_LINE as u64 - 1)), DC_LINE)) + } + CACH_PI if IC_LINE != 0 && tag & PI_VALID != 0 => { + Some((pa_hi | ((tag >> 8) & 0xff_ffff) << 12 | (va & 0xfff & !(IC_LINE as u64 - 1)), IC_LINE)) + } + CACH_SD if L2_LINE != 0 && (tag >> SD_STATE_SHIFT) & 3 != 0 => { + Some((pa_hi | ((tag >> 14) & 0x3_ffff) << 18 | (va & 0x3_ffff & !(L2_LINE as u64 - 1)), L2_LINE)) + } + _ => None, + } + } + + /// An index operand that names slot `i` of `sel`: `tag_slot`'s inverse. + fn slot_va(&self, sel: u32, i: usize) -> u64 { + let line = match sel { CACH_PI => IC_LINE, CACH_PD => DC_LINE, _ => L2_LINE }; + if Self::R5K && matches!(sel, CACH_PI | CACH_PD) { + let sets = if sel == CACH_PI { Self::IC_LINES } else { Self::DC_LINES } / 2; + return ((i / sets) << 14 | (i % sets) * line) as u64; + } + ((i / WAYS) * line + (i % WAYS)) as u64 + } + + /// A CACHE instruction changed slot `idx` of `sel`'s tag from `old` to + /// `new`; `va` is the operand that named it. A held line that stops being + /// valid, or moves, gives its page back; one that becomes valid takes one. + /// The tags stay the only record of which lines are valid. + #[cold] + #[inline(never)] + fn tag_changes(&self, sel: u32, idx: usize, va: u64, old: u64, new: u64) { + let Some(sp) = self.view_space() else { return }; + let now = self.line_of(sel, new, va); + if self.shadow(sel).held[idx] { + let was = self.line_of(sel, old, va).expect("a held line is valid"); + if now == Some(was) { + return; // the state changed, not the line + } + self.line_leaves(sp, sel, idx, was); + } + if let Some(line) = now { + self.line_enters(sp, sel, idx, va, line); + } + } + + /// Slot `idx` of `sel` holds a valid line for `[pa, pa + len)`: its page + /// becomes private in the CPU's view and the line's bytes are written + /// into it from the data array, since Index Store Tag writes the tag + /// alone (NEC VR5000/VR10000 instruction manual pp. 24-25). A line with + /// no memory behind it holds no page and stays in the arrays, as every + /// line did before. + fn line_enters(&self, sp: &PpMemSpace, sel: u32, idx: usize, va: u64, (pa, len): (u64, usize)) { + let Some(new) = sp.hold_line_page(self.view_no(), pa) else { return }; + let base = unsafe { *self.tc_base.get() }; + if l2_op_log(CACHE_LOG_OP) { + crate::dlog!(LogModule::L2c, "shadow: -> line {pa:#x} holds its page (new: {new})"); + } + let s = self.shadow(sel); + s.held[idx] = true; + if s.data.is_empty() { + return; // the R5000 has no Index Store Data: the page starts as memory + } + let (way, first) = (va & 1, va & !(len as u64 - 1)); + let unit = if sel == CACH_SD { 8 } else { 4 }; + for k in 0..len / unit { + let word = s.data[self.data_slot(sel, s.data.len(), first + (unit * k) as u64 + way)]; + let at = pa + (unit * k) as u64; + match sel { + CACH_PI => self.put_instruction(at, word), + CACH_PD => unsafe { *(base.add(at as usize) as *mut u32) = word as u32 }, + // TagHi is the doubleword's high word and TagLo its low word + // (MIPS R10000 UM p. 201; NEC VR10000 UM p. 289, Fig. 13-33). + // Neither manual says which address each word belongs to. + // TagLo at the lower address is inferred from shipped PROMs: + // PROM 6.210 stages code pairs with the word at A in TagLo + // and the one at A + 4 in TagHi, and its staged code equals + // its image only this way, as does the IP35 boot-flash + // PROM's staging. The window keeps a doubleword's words in + // address order, low half first, so this is the data as is. + _ => unsafe { *(base.add(at as usize) as *mut u64) = word }, + } + } + if sel == CACH_SD { + self.set_refill(s.tags[idx], pa, len, true); + } + } + + /// The line in slot `idx` of `sel` leaves, unwritten, as Index Store Tag + /// leaves it. With the page's last line the view shows memory there + /// again. Until then the line's bytes stay in the page. + fn line_leaves(&self, sp: &PpMemSpace, sel: u32, idx: usize, (pa, len): (u64, usize)) { + let s = self.shadow(sel); + s.held[idx] = false; + if sel == CACH_SD { + self.set_refill(s.tags[idx], pa, len, false); + } + let gone = sp.release_line_page(self.view_no(), pa); + if l2_op_log(CACHE_LOG_OP) { + crate::dlog!(LogModule::L2c, "shadow: -> line {pa:#x} leaves (page released: {gone})"); + } + } + + /// Index Store Data into a held line writes through to its page, so the + /// next cached access sees it. Same quadword rule as the data array. + fn store_into_held_line(&self, sel: u32, va: u64, value: u64) { + let idx = self.tag_slot(sel, va); + let Some((pa, len)) = self.line_of(sel, self.shadow(sel).tags[idx], va) else { return }; + let unit = if sel == CACH_SD { 8 } else { 4 }; + let at = pa + (va & (len as u64 - 1) & !(unit - 1)); + let base = unsafe { *self.tc_base.get() }; + match sel { + CACH_PI => self.put_instruction(at, value & 0xf_ffff_ffff), + CACH_PD => unsafe { *(base.add(at as usize) as *mut u32) = value as u32 }, + _ => unsafe { + *(base.add(at as usize) as *mut u64) = value; + *(base.add((at ^ 8) as usize) as *mut u64) = 0; + }, + } + if sel == CACH_SD { + self.set_refill(self.shadow(sel).tags[idx], pa, len, true); + } + } + + /// A primary instruction word goes into its line's page as the + /// instruction it predecodes, so a fetch runs it. A word of a class the + /// model does not know leaves the page as it was, where hardware would + /// run it. + fn put_instruction(&self, at: u64, word: u64) { + match l1i_decode(word) { + Ok(f) => unsafe { *((*self.tc_base.get()).add(at as usize) as *mut u32) = f }, + Err(class) => crate::dlog!(LogModule::L2c, + "shadow: primary instruction word {word:#011x} at {at:#x}: predecode class {class:#04x} unknown, memory's word runs"), + } + } + + /// A held secondary line `[pa, pa + len)` with tag `tag` is what a primary + /// instruction refill in each set it maps to would load, or stops being. + /// The set is the line's virtual index, TagLo[8:7] (VA[13:12], NEC + /// U10278EJ3V0UM p. 287), over PA[11:6]. Two lines for one set: the + /// later wins. + fn set_refill(&self, tag: u64, pa: u64, len: usize, held: bool) { + let refill = unsafe { &mut *self.refill.get() }; + if !held { + // By address: the tag may already name another line. + refill.iter_mut().filter(|r| (pa..pa + len as u64).contains(*r)).for_each(|r| *r = u64::MAX); + return; + } + for half in (pa..pa + len as u64).step_by(IC_LINE.max(1)) { + if let Some(r) = refill.get_mut(((tag >> 7 & 3) << 12 | (half & 0xfff)) as usize / IC_LINE) { + *r = half; + } + } + } + + /// Index Load Data (I) in a set a held secondary line maps to, which + /// Index Store Data (I) has not written since and where no held primary + /// line is: the word a refill from that + /// line would have left, predecoded. The model never refills on a fetch, + /// so this assumes the line was refilled, whichever way the operand + /// names. The Tezro's PROM runs a staged secondary line and reads the + /// primary words back to stage them itself. Index Load Tag (I) still + /// reads the stored tag. + #[cold] + #[inline(never)] + fn refilled_word(&self, va: u64) -> Option { + let half = *unsafe { &*self.refill.get() }.get((va as usize / IC_LINE.max(1)) % Self::PI_SETS.max(1))?; + if half == u64::MAX || self.shadow(CACH_PI).held[self.tag_slot(CACH_PI, va)] { + return None; + } + let at = half + (va & (IC_LINE as u64 - 1) & !3); + Some(l1i_encode(unsafe { *((*self.tc_base.get()).add(at as usize) as *const u32) })) + } + + /// How many lines hold pages. + fn held_count(&self) -> usize { + [CACH_PI, CACH_PD, CACH_SD].iter().map(|&sel| self.shadow(sel).held.iter().filter(|&&h| h).count()).sum() + } + + /// Every held line goes, with no writeback: power-on, or a snapshot + /// restore. + fn drop_held_lines(&self) { + let Some(sp) = self.view_space() else { return }; + for sel in [CACH_PI, CACH_PD, CACH_SD] { + for i in 0..self.shadow(sel).held.len() { + if self.shadow(sel).held[i] { + let line = self.line_of(sel, self.shadow(sel).tags[i], self.slot_va(sel, i)).expect("a held line is valid"); + self.line_leaves(sp, sel, i, line); + } + } + } } } @@ -417,8 +729,19 @@ impl< fn tcache_base_ptr(&self) -> *mut u8 { unsafe { *self.tc_base.get() } } - fn set_pd_tag_store_hook(&self, hook: Box) { - unsafe { *self.pd_tag_hook.get() = Some(hook) }; + unsafe fn set_cached_view(&self, sp: *const PpMemSpace, view: usize) { + unsafe { + *self.view_space.get() = sp; + *self.view.get() = view; + } + } + + /// `l2 probe`: the lines holding pages in the CPU's cached view. + fn debug_probe(&self, _cache_name: &str, _virt_addr: u64, _phys_addr: u64) -> String { + let pages = self.view_space().map(|sp| sp.held_line_pages(self.view_no())).unwrap_or_default(); + let list: Vec = pages.iter().map(|p| format!("{p:#x}")).collect(); + format!("shadow: {} valid lines hold {} pages in the cached view [{}]", + self.held_count(), pages.len(), list.join(" ")) } /// Tags only: a fetch reads memory through the decode cache and changes @@ -565,12 +888,6 @@ impl< let sel = cache_op & 3; let op = cache_op & 0x1C; - if op == C_IST && sel == CACH_PD { - if let Some(hook) = unsafe { &*self.pd_tag_hook.get() } { - hook(phys_addr as u32); - } - } - // Tag operations under `op`, the whole data-array walk only under // `hit` as well — see `l2_op_log`. if l2_op_log(if matches!(op, C_IST | C_ILT) { CACHE_LOG_OP } else { CACHE_LOG_HIT }) { @@ -589,7 +906,13 @@ impl< let idx = self.tag_slot(sel, virt_addr); let s = self.shadow(sel); if op == C_IST { - s.tags[idx] = r4k_store_taglo(Self::R5K, sel, phys_addr as u32) as u64; + let old = std::mem::replace(&mut s.tags[idx], r4k_store_taglo(Self::R5K, sel, phys_addr as u32) as u64); + // The O2's POST keeps its stack in R5000 primary data lines + // as it does in R10000 ones. An instruction line, with no + // Index Store Data to fill it, could hold only memory. + if Self::R5K && sel == CACH_PD && old != s.tags[idx] { + self.tag_changes(sel, idx, virt_addr, old, s.tags[idx]); + } return 0; } return r4k_load_taglo(Self::R5K, sel, s.tags[idx] as u32) as u64; @@ -619,7 +942,11 @@ impl< // here destroyed real tag bits. let set = (idx / WAYS).min(s.mru.len() - 1); s.mru[set] = phys_addr & MRU_BIT != 0; + let old = s.tags[idx]; s.tags[idx] = phys_addr & mask & !MRU_BIT; + if Self::R10K && matches!(sel, CACH_PI | CACH_PD | CACH_SD) && old != s.tags[idx] { + self.tag_changes(sel, idx, virt_addr, old, s.tags[idx]); + } } 0 } @@ -654,6 +981,11 @@ impl< crate::dlog!(LogModule::L2c, "shadow: -> ILD slot={slot} data={:#018x} ecc={:#x}", s.data[slot], s.ecc[slot]); } + if sel == CACH_PI { + if let Some(p) = self.refilled_word(virt_addr) { + return p; + } + } s.data[slot] } } @@ -666,7 +998,8 @@ impl< let idx = self.tag_slot(sel, virt_addr); let s = self.shadow(sel); if idx < s.tags.len() { - s.tags[idx] = 0; + let old = std::mem::take(&mut s.tags[idx]); + self.tag_changes(sel, idx, virt_addr, old, 0); } 0 } @@ -680,6 +1013,7 @@ impl< /// Power-on: tags read back invalid, and nothing decoded survives — RAM /// may be reloaded without a write counter moving. fn power_on(&self) { + self.drop_held_lines(); for sel in [CACH_PI, CACH_PD, CACH_SD] { let s = self.shadow(sel); s.tags.fill(0); @@ -688,10 +1022,27 @@ impl< unsafe { (*self.decode.get()).flush() }; } - /// A restore replaces RAM wholesale; drop every decoded instruction. The - /// shadow arrays are not snapshotted (nothing reads them but diagnostics). - fn load_cache_state(&self, _v: &toml::Value) -> Result<(), String> { + /// The shadow arrays are not snapshotted (nothing reads them but + /// diagnostics). Lines holding pages are, as a count: a PROM's cache + /// phase is the only time any are held, and a snapshot of it would + /// restore without them, so it is refused. + fn save_cache_state(&self) -> toml::Value { + let mut t = toml::map::Map::new(); + let n = self.held_count(); + if n > 0 { + t.insert("held_lines".into(), toml::Value::Integer(n as i64)); + } + toml::Value::Table(t) + } + + /// A restore replaces RAM wholesale; drop every decoded instruction and + /// every held line. + fn load_cache_state(&self, v: &toml::Value) -> Result<(), String> { + if let Some(n) = crate::snapshot::get_field(v, "held_lines") { + return Err(format!("cache: the snapshot was taken with {n} cache lines holding memory, which it does not carry")); + } unsafe { (*self.decode.get()).flush() }; + self.drop_held_lines(); Ok(()) } @@ -1084,20 +1435,6 @@ mod tests { (::PRID, ::FIR, ::MIPS4, ::TLB_ENTRIES, ::VA_BITS, ::R10K_CACHE_OPS)); } - /// The tag store hook sees primary data tag stores and nothing else. - #[test] - fn the_tag_store_hook_sees_only_primary_data_tag_stores() { - let c = cache(); - let seen = Arc::new(std::sync::Mutex::new(Vec::new())); - let s = seen.clone(); - c.set_pd_tag_store_hook(Box::new(move |tag| s.lock().unwrap().push(tag))); - c.cache_op(C_IST | CACH_PI, 0x8000_0000, 0x11); - c.cache_op(C_IST | CACH_SD, 0x8000_0000, 0x22); - c.cache_op(C_ILT | CACH_PD, 0x8000_0000, 0x33); - c.cache_op(C_IST | CACH_PD, 0x8000_0000, 0xc8); - assert_eq!(*seen.lock().unwrap(), vec![0xc8]); - } - /// Index_Store_Tag then Index_Load_Tag returns exactly what the full model /// returns, for random TagLo words at random indexes: the shadow must not /// be observably different to a PROM, IDE or IRIX reading tags. Both @@ -1144,4 +1481,327 @@ mod tests { assert!(g.supported && g.tagless && !g.has_l2, "tagless, and no L2 decode slots for a store to invalidate: {g:?}"); } + + // ---- valid lines hold pages in the CPU's cached view ---- + + use crate::ppmem::{MappedMemory, PpMemory}; + + /// A cache reading memory through a cached view of a fresh window, with + /// a 64 MB bank mapped at 0. Its bus is the bank itself: memory as the + /// uncached path sees it. Boxed: the view publishes into the cache's + /// inline bitmap field. + fn windowed() -> (Box, Box, Arc) { + let (space, mut banks) = PpMemSpace::with_bank_sizes(&[64]).unwrap(); + let space = Box::new(space); + let bank = Arc::new(banks.remove(0)); + let c = Box::new(R10000ShadowCache::from(bank.clone() as Arc)); + attach(&c, &space); + space.map_bank(0, 0, 64 << 20, 64 << 20).unwrap(); + (c, space, bank) + } + + fn attach(c: &R10000ShadowCache, space: &PpMemSpace) -> usize { + let v = space.add_cached_view(&[c.tcache_bitmap_ptr()]).unwrap(); + unsafe { + c.set_tcache_window(space.view_base(v)); + c.set_cached_view(space, v); + } + v + } + + /// TagLo of a DirtyExclusive primary data line for `pa`'s page. + fn pd_dirty(pa: u64) -> u64 { (pa >> 12) << 8 | 3 << PD_STATE_SHIFT } + + /// TagLo of a secondary line for `pa` in `state`. + fn sd_tag(pa: u64, state: u64) -> u64 { (pa >> 18) << 14 | state << SD_STATE_SHIFT } + + /// The O2's POST: a valid tag for page 0, the page tested uncached while + /// the cached side keeps its stack, and an invalid tag that drops the + /// line without a writeback. + #[test] + fn a_valid_line_gives_the_cached_side_its_own_page() { + let (c, space, bank) = windowed(); + bank.write32(0x40, 0x1111_1111); + c.cache_op(C_R10K_ISD | CACH_PD, 0x8000_0040, 0x1111_1111); + c.cache_op(C_IST | CACH_PD, 0x8000_0040, pd_dirty(0)); + assert_eq!(space.held_line_pages(0), vec![0]); + assert_eq!(c.read::<4>(0, 0x40).data, 0x1111_1111); + c.write::<4>(0, 0x40, 0x2222_2222); + bank.write32(0x40, 0x3333_3333); + assert_eq!(c.read::<4>(0, 0x40).data, 0x2222_2222, "cached: the line"); + assert_eq!(bank.read32(0x40).data, 0x3333_3333, "uncached: memory"); + c.cache_op(C_IST | CACH_PD, 0x8000_0040, 0); + assert!(space.held_line_pages(0).is_empty()); + assert_eq!(c.read::<4>(0, 0x40).data, 0x3333_3333, "dropped, not written back"); + } + + /// Index Store Tag writes the tag alone, so the line holds what Index + /// Store Data put in the data array before, and Index Store Data after + /// writes through. Index Load Data reads the array. + #[test] + fn a_line_holds_its_data_array_and_takes_later_stores() { + let (c, _space, bank) = windowed(); + bank.write32(0x2004, 0x5555_5555); + c.cache_op(C_R10K_ISD | CACH_PD, 0x8000_0004, 0xdead_beef); + c.cache_op(C_IST | CACH_PD, 0x8000_0000, pd_dirty(0x2000)); + assert_eq!(c.read::<4>(0, 0x2004).data, 0xdead_beef); + c.cache_op(C_R10K_ISD | CACH_PD, 0x8000_000c, 0x0123_4567); + assert_eq!(c.read::<4>(0, 0x200c).data, 0x0123_4567); + assert_eq!(c.cache_op(C_R10K_ILD | CACH_PD, 0x8000_000c, 0), 0x0123_4567); + assert_eq!(bank.read32(0x2004).data, 0x5555_5555, "memory never saw the line"); + } + + /// A secondary line takes a doubleword at a time with TagLo at the lower + /// address. That is how the shipped PROMs stage code: the instruction at + /// A in TagLo and the one at A + 4 in TagHi, then Index Store Data (S), + /// and cached accesses see them in that order. Index Store Data (S) also + /// zeroes the other doubleword of its quadword (R10000 UM p. 202). + #[test] + fn a_secondary_line_runs_code_staged_the_way_the_proms_stage_it() { + let (c, space, _bank) = windowed(); + let (at_a, at_a4) = (0x2404_0002u64, 0x03e0_0008u64); // addiu $4,$0,2; jr $31 + c.cache_op(C_R10K_ISD | CACH_SD, 0x8000_3008, at_a4 << 32 | at_a); + c.cache_op(C_IST | CACH_SD, 0x8000_3000, sd_tag(0x3000, 2)); + assert_eq!(space.held_line_pages(0), vec![0x3000]); + assert_eq!(c.read::<4>(0, 0x3008).data, at_a, "the word at A"); + assert_eq!(c.read::<4>(0, 0x300c).data, at_a4, "the word at A + 4"); + c.cache_op(C_R10K_ISD | CACH_SD, 0x8000_3000, 0x9999); + assert_eq!(c.read::<8>(0, 0x3000).data, 0x9999 << 32); + assert_eq!(c.read::<8>(0, 0x3008).data, 0, "the partner doubleword is zeroed"); + } + + /// A tag test's patterns name no memory: they hold nothing, and stay + /// in the arrays as before. + #[test] + fn a_line_with_no_memory_behind_holds_no_page() { + let (c, space, _bank) = windowed(); + c.cache_op(C_IST | CACH_SD, 0x8000_0000, sd_tag(0x55_5554_0000, 3) | 0x5); + c.cache_op(C_IST | CACH_PD, 0x8000_0000, pd_dirty(0x8000_0000)); + assert!(space.held_line_pages(0).is_empty()); + assert_eq!(c.cache_op(C_ILT | CACH_SD, 0x8000_0000, 0) & 0xffff_ffff, (sd_tag(0x55_5554_0000, 3) | 0x5) & 0xffff_ffff); + } + + /// The R4400 PROMs keep nothing in their caches: a valid tag there holds + /// no page, and none of this runs for them. + #[test] + fn an_r4400_tag_holds_nothing() { + let (space, mut banks) = PpMemSpace::with_bank_sizes(&[64]).unwrap(); + let bank = Arc::new(banks.remove(0)); + let c = Box::new(R4400ShadowCache::from(bank as Arc)); + let v = space.add_cached_view(&[c.tcache_bitmap_ptr()]).unwrap(); + unsafe { + c.set_tcache_window(space.view_base(v)); + c.set_cached_view(&space, v); + } + space.map_bank(0, 0, 64 << 20, 64 << 20).unwrap(); + c.cache_op(C_IST | CACH_PD, 0x8000_2000, 0x2 << 8 | 3 << 6); + c.cache_op(C_IST | CACH_SD, 0x8000_2000, 0x1 << 13 | 5 << 10); + assert!(space.held_line_pages(v).is_empty()); + } + + /// An R5000 cache on a cached view of a fresh window with a 64 MB bank + /// at 0, as `windowed` builds an R10000 one. + fn r5000_windowed() -> (Box, Box, Arc) { + let (space, mut banks) = PpMemSpace::with_bank_sizes(&[64]).unwrap(); + let space = Box::new(space); + let bank = Arc::new(banks.remove(0)); + let c = Box::new(R5000ShadowCache::from(bank.clone() as Arc)); + let v = space.add_cached_view(&[c.tcache_bitmap_ptr()]).unwrap(); + unsafe { + c.set_tcache_window(space.view_base(v)); + c.set_cached_view(&*space, v); + } + space.map_bank(0, 0, 64 << 20, 64 << 20).unwrap(); + (c, space, bank) + } + + /// The O2's POST on the R5000, as the shipped PROM runs it on iris: + /// Index Store Tag (D) with TagLo 0xc0 (page 0, Valid) at every index of + /// 0x8000_0000..=0x8000_1000, the stack stored cached, page 0 tested + /// uncached, then an invalid tag at each index. The cached side keeps + /// the stack until the last line goes, which drops it unwritten. The + /// R5000 has no Index Store Data, so the page starts as memory. + #[test] + fn an_r5000_primary_data_line_gives_the_cached_side_its_own_page() { + let (c, space, bank) = r5000_windowed(); + bank.write32(0xff4, 0x1111_1111); + for va in (0x8000_0000..=0x8000_1000u64).step_by(32) { + c.cache_op(C_IST | CACH_PD, va, 0xc0); + } + assert_eq!(space.held_line_pages(0), vec![0]); + assert_eq!(c.read::<4>(0, 0xff4).data, 0x1111_1111, "the page starts as memory"); + c.write::<4>(0, 0xff4, 0xbfc0_4400); + bank.write32(0xff4, 0xa000_0ff0); + assert_eq!(c.read::<4>(0, 0xff4).data, 0xbfc0_4400, "cached: the stack"); + assert_eq!(bank.read32(0xff4).data, 0xa000_0ff0, "uncached: the memory test"); + assert_eq!(c.cache_op(C_ILT | CACH_PD, 0x8000_0fe0, 0) & 0xc0, 0xc0, "the tag reads back"); + for va in (0x8000_0020..=0x8000_1000u64).step_by(32) { + c.cache_op(C_IST | CACH_PD, va, 0); + } + assert_eq!(space.held_line_pages(0), vec![0], "until its last line goes"); + c.cache_op(C_IST | CACH_PD, 0x8000_0000, 0); + assert!(space.held_line_pages(0).is_empty()); + assert_eq!(c.read::<4>(0, 0xff4).data, 0xa000_0ff0, "dropped, not written back"); + } + + /// The R5000's second way, VA[14], holds a page like the first, and + /// power-on drops it. Instruction lines hold nothing: with no Index + /// Store Data, one could hold only what memory holds. There is no + /// secondary. + #[test] + fn an_r5000_line_in_either_way_holds_and_power_on_drops_it() { + let (c, space, _bank) = r5000_windowed(); + c.cache_op(C_IST | CACH_PI, 0x8000_0000, 0x3 << 8 | 0x80); + c.cache_op(C_IST | CACH_SD, 0x8000_0000, 0x1 << 13 | 5 << 10); + assert!(space.held_line_pages(0).is_empty()); + c.cache_op(C_IST | CACH_PD, 0x8000_4fe0, 0x5 << 8 | 0xc0); + assert_eq!(space.held_line_pages(0), vec![0x5000]); + assert_eq!(c.cache_op(C_ILT | CACH_PD, 0x8000_0fe0, 0) & 0xc0, 0, "way 0 is untouched"); + c.power_on(); + assert!(space.held_line_pages(0).is_empty()); + assert!(c.debug_probe("l2", 0, 0).contains("0 valid lines hold 0 pages")); + } + + /// The Tezro: two CPUs keep their stacks in lines for the same physical + /// page, with different contents, and neither sees the other's. + #[test] + fn two_cpus_hold_the_same_page_apart() { + let (space, mut banks) = PpMemSpace::with_bank_sizes(&[64]).unwrap(); + let space = Box::new(space); + let bank = Arc::new(banks.remove(0)); + let a = Box::new(R10000ShadowCache::from(bank.clone() as Arc)); + let b = Box::new(R10000ShadowCache::from(bank.clone() as Arc)); + attach(&a, &space); + attach(&b, &space); + space.map_bank(0, 0, 64 << 20, 64 << 20).unwrap(); + bank.write32(0x10_0040, 0x1234_5678); + for c in [&a, &b] { + c.cache_op(C_R10K_ISD | CACH_PD, 0x8000_0040, 0x1234_5678); + c.cache_op(C_IST | CACH_PD, 0x8000_0040, pd_dirty(0x10_0000)); + } + a.write::<4>(0, 0x10_0040, 0xaaaa_aaaa); + b.write::<4>(0, 0x10_0040, 0xbbbb_bbbb); + assert_eq!(a.read::<4>(0, 0x10_0040).data, 0xaaaa_aaaa); + assert_eq!(b.read::<4>(0, 0x10_0040).data, 0xbbbb_bbbb); + assert_eq!(bank.read32(0x10_0040).data, 0x1234_5678, "memory is neither"); + assert_eq!(unsafe { *(space.window_base().add(0x10_0040) as *const u32) }, 0x1234_5678, + "the shared window, which other masters read, is memory"); + a.cache_op(C_IST | CACH_PD, 0x8000_0040, 0); + assert_eq!(a.read::<4>(0, 0x10_0040).data, 0x1234_5678); + assert_eq!(b.read::<4>(0, 0x10_0040).data, 0xbbbb_bbbb); + } + + /// Two lines of one page: the page stays until the last goes. The R12000 + /// zeroing a primary data tag at Index Writeback Invalidate ends a line + /// like an invalid tag store. + #[test] + fn a_page_goes_with_its_last_line() { + let (space, mut banks) = PpMemSpace::with_bank_sizes(&[64]).unwrap(); + let space = Box::new(space); + let bank = Arc::new(banks.remove(0)); + let c = Box::new(R12000ShadowCache::from(bank.clone() as Arc)); + let v = space.add_cached_view(&[c.tcache_bitmap_ptr()]).unwrap(); + unsafe { + c.set_tcache_window(space.view_base(v)); + c.set_cached_view(&*space, v); + } + space.map_bank(0, 0, 64 << 20, 64 << 20).unwrap(); + bank.write32(0x6000, 0x600d); + c.cache_op(C_IST | CACH_PD, 0x8000_6000, pd_dirty(0x6000)); + c.cache_op(C_IST | CACH_PD, 0x8000_6040, pd_dirty(0x6000)); + c.write::<4>(0, 0x6000, 0xbad); + c.cache_op(C_IST | CACH_PD, 0x8000_6000, 0); + assert_eq!(space.held_line_pages(v), vec![0x6000]); + assert_eq!(c.read::<4>(0, 0x6000).data, 0xbad, "the page stays the view's"); + c.cache_op(C_IWBINV | CACH_PD, 0x8000_6040, 0); + assert!(space.held_line_pages(v).is_empty()); + assert_eq!(c.read::<4>(0, 0x6000).data, 0x600d); + } + + /// Power-on drops every held page. A snapshot taken while lines hold + /// pages is refused at restore. + #[test] + fn power_on_drops_held_pages_and_a_snapshot_of_them_is_refused() { + let (c, space, _bank) = windowed(); + c.cache_op(C_IST | CACH_PD, 0x8000_0000, pd_dirty(0x7000)); + let saved = c.save_cache_state(); + assert!(c.load_cache_state(&saved).is_err()); + c.power_on(); + assert!(space.held_line_pages(0).is_empty()); + assert!(c.debug_probe("l2", 0, 0).contains("0 valid lines hold 0 pages")); + assert!(c.load_cache_state(&c.save_cache_state()).is_ok()); + } + + // ---- primary instruction lines ---- + + /// Every class of the table, encoded and decoded back, one field moved + /// at a time for the example word, and what the table refuses. + #[test] + fn predecoded_words_round_trip() { + let p = 0x3_4420_0004; // addiu $4,$0,2 + for (q, f) in [(p, 0x2404_0002), (p | 0x1f << 5, 0x2404_f802), (p | 0x1f << 10, 0x27e4_0002), (p | 0x1f << 15, 0x2404_07c2)] { + assert_eq!(l1i_decode(q), Ok(f), "{q:#x}"); + } + for (class, f) in [(0x08, 0x0009_20f8), (0x08, 0x0009_20fc), (0x18, 0x0109_5025), (0x1a, 0x2084_0010), + (0x1a, 0x3484_0010), (0x1a, 0x3c04_8000), (0x1e, 0x6484_0010), (0x35, 0xbd04_0040), + (0x36, 0xdd04_0008), (0x3f, 0x4084_6000), (0x40, 0x0c00_1234), (0x41, 0x03e0_0008), + (0x41, 0x0800_1234), (0x71, 0x0481_0004), (0, 0)] { + assert_eq!((l1i_encode(f) >> 29, l1i_decode(l1i_encode(f))), (class, Ok(f)), "{f:#x}"); + } + for f in [0x0009_20c0, 0x2884_0010, 0x4422_3000] { // sll, slti, mtc1: no class + assert_eq!((l1i_encode(f), l1i_decode(L1I_RAW | f as u64)), (L1I_RAW | f as u64, Ok(f))); + } + assert_eq!((l1i_decode(1), l1i_decode(0x2a << 29)), (Err(0), Err(0x2a))); + } + + /// TagLo of a valid primary instruction line for `pa`'s page. + fn pi_valid(pa: u64) -> u64 { (pa >> 12) << 8 | PI_VALID } + + /// A valid primary instruction line puts the instructions its words + /// predecode into the page, leaving memory's word for one it cannot + /// decode, and Index Store Data (I) writes through. + #[test] + fn a_primary_instruction_line_puts_its_instructions_in_the_page() { + let (c, space, bank) = windowed(); + bank.write32(0x9008, 0x1111_1111); + let va = 0x8000_9001u64; // way 1 + c.cache_op(C_R10K_ISD | CACH_PI, va, 0x3_4420_0004); + c.cache_op(C_R10K_ISD | CACH_PI, va + 8, 0x2a << 29); + c.cache_op(C_IST | CACH_PI, va, pi_valid(0x9000)); + assert_eq!(space.held_line_pages(0), vec![0x9000]); + assert_eq!([0x9000, 0x9008].map(|a| c.read::<4>(0, a).data), [0x2404_0002, 0x1111_1111]); + c.cache_op(C_R10K_ISD | CACH_PI, va + 4, l1i_encode(0x03e0_0008)); + assert_eq!(c.read::<4>(0, 0x9004).data, 0x03e0_0008); + assert_eq!(c.cache_op(C_R10K_ILD | CACH_PI, va, 0), 0x3_4420_0004, "Index Load Data (I) reads the array"); + assert_eq!(bank.read32(0x9000).data, 0, "memory never saw it"); + c.cache_op(C_IST | CACH_PI, va, 0); + assert!(space.held_line_pages(0).is_empty()); + } + + /// A held secondary line answers Index Load Data (I) in the primary sets + /// it maps to, as the predecoded words a refill would leave, until Index + /// Store Data (I) writes the set or the line goes. A held primary line + /// answers for itself. + #[test] + fn index_load_data_i_reads_a_refill_until_the_guest_writes_the_set() { + let (c, _space, _bank) = windowed(); + c.cache_op(C_IST | CACH_SD, 0x8000_4f00, sd_tag(0x4f00, 2) | 1 << 7); // VA[13:12] = 1 + c.cache_op(C_R10K_ISD | CACH_SD, 0x8000_4f10, 0x0140_0008); // jr $10 at 0x4f10 + let set = |off: u64| 0x8000_1f00 | off; + for va in [set(0x10), set(0x10) | 1] { + assert_eq!(c.cache_op(C_R10K_ILD | CACH_PI, va, 0), l1i_encode(0x0140_0008), "either way"); + } + assert_eq!(c.cache_op(C_R10K_ILD | CACH_PI, 0x8000_0f10, 0), 0, "another set: the array"); + c.cache_op(C_R10K_ISD | CACH_PI, set(0x10), 0x5_5555_5555); + assert_eq!(c.cache_op(C_R10K_ILD | CACH_PI, set(0x10), 0), 0x5_5555_5555, "the guest's pattern"); + c.cache_op(C_R10K_ISD | CACH_SD, 0x8000_4f10, 0x0140_0008); + assert_eq!(c.cache_op(C_R10K_ILD | CACH_PI, set(0x10), 0), l1i_encode(0x0140_0008), "restaged: a refill again"); + c.cache_op(C_R10K_ISD | CACH_PI, set(0x10), 0x3_4420_0004); + c.cache_op(C_IST | CACH_PI, set(0x10), pi_valid(0x1_4000)); + c.cache_op(C_R10K_ISD | CACH_SD, 0x8000_4f10, 0x0140_0008); + assert_eq!(c.cache_op(C_R10K_ILD | CACH_PI, set(0x10), 0), 0x3_4420_0004, "a held primary line answers for itself"); + c.cache_op(C_IST | CACH_PI, set(0x10), 0); + c.cache_op(C_IST | CACH_SD, 0x8000_4f00, 0); + assert_eq!(c.cache_op(C_R10K_ILD | CACH_PI, set(0x10), 0), 0x3_4420_0004, "gone with the line: the array"); + } } diff --git a/src/cpu/mips_cache_v2.rs b/src/cpu/mips_cache_v2.rs index 2b92dc849..5619a9cb3 100644 --- a/src/cpu/mips_cache_v2.rs +++ b/src/cpu/mips_cache_v2.rs @@ -669,12 +669,6 @@ pub trait MipsCache: Send + Sync { /// Share the L1-I hit/fetch counters with the status display. No-op where absent. fn set_l1i_counters(&mut self, _hit: Arc, _fetch: Arc) {} - /// Have `hook` called with TagLo on every Index Store Tag to the primary - /// data cache: a machine whose firmware builds memory out of cache tags - /// (the O2's POST stack, see `O2Memory::stack_tag_store`). No-op where - /// absent. Set once, before the CPU runs. - fn set_pd_tag_store_hook(&self, _hook: Box) {} - /// tcache: point the cache at ppmem's window and mapped-region bitmap, so /// cacheable RAM accesses read/write RAM directly instead of copying line /// data (docs/tcache-design.md). Default no-op — a cache with no @@ -684,6 +678,15 @@ pub trait MipsCache: Send + Sync { /// `base` must be ppmem's window base and `bitmap` a live `u64` that /// outlives this cache. unsafe fn set_tcache_window(&self, _base: *mut u8) {} + + /// The cached view this cache reads memory through + /// (`PpMemSpace::add_cached_view`), as `sp` and its number, so valid + /// lines can hold their pages there. Set with the window, before the CPU + /// runs. No-op where the cache holds no lines. + /// + /// # Safety + /// `sp` must outlive the cache. + unsafe fn set_cached_view(&self, _sp: *const crate::ppmem::PpMemSpace, _view: usize) {} /// The ppmem window's `window_shift`, when it is not the 4GB window's. fn set_tcache_shift(&self, _shift: u32) {} diff --git a/src/cpu/mips_exec.rs b/src/cpu/mips_exec.rs index 782883bf7..fb2c35136 100644 --- a/src/cpu/mips_exec.rs +++ b/src/cpu/mips_exec.rs @@ -2872,6 +2872,14 @@ impl MipsExecutor { unsafe { self.cache.set_tcache_window(base) }; } + /// See `MipsCache::set_cached_view`. + /// + /// # Safety + /// `sp` must outlive the executor. + pub unsafe fn set_cached_view(&self, sp: *const crate::ppmem::PpMemSpace, view: usize) { + unsafe { self.cache.set_cached_view(sp, view) }; + } + /// tcache: the window's `ppmem::window_shift`, when it is not 4GB's. pub fn set_tcache_shift(&self, shift: u32) { self.cache.set_tcache_shift(shift); @@ -10561,7 +10569,7 @@ impl MipsCpu { } /// ppmem: pointer to the executor's inline `ppmem_bitmap` field, for - /// `PpMemSpace::set_bitmap_sink`. Fixed for the life of this `MipsCpu` — + /// `PpMemSpace::add_cached_view`. Fixed for the life of this `MipsCpu` — /// taken once in `new()` after the executor reached its final address. pub fn ppmem_bitmap_ptr(&self) -> *mut u64 { self.ppmem_bitmap_ptr @@ -10575,6 +10583,14 @@ impl MipsCpu { unsafe { self.executor.lock().set_tcache_window(base) }; } + /// See `MipsExecutor::set_cached_view`. + /// + /// # Safety + /// `sp` must outlive the CPU. + pub unsafe fn set_cached_view(&self, sp: *const crate::ppmem::PpMemSpace, view: usize) { + unsafe { self.executor.lock().set_cached_view(sp, view) }; + } + /// tcache: see `MipsExecutor::set_tcache_shift`. pub fn set_tcache_shift(&self, shift: u32) { self.executor.lock().set_tcache_shift(shift); @@ -11293,13 +11309,18 @@ pub trait CpuDevice: Device + Resettable + Saveable + Send + Sync { fn cycles_ptr(&self) -> crate::cpu::mips_core::CyclesPtr; fn interrupts_ptr(&self) -> *const AtomicU64; /// ppmem: the executor's inline `ppmem_bitmap` field, for - /// `PpMemSpace::set_bitmap_sink`. + /// `PpMemSpace::add_cached_view`. fn ppmem_bitmap_ptr(&self) -> *mut u64; /// tcache: hand the cache ppmem's window base + mapped-region bitmap. /// /// # Safety /// Same contract as `CpuCache::set_tcache_window`. unsafe fn set_tcache_window(&self, base: *mut u8); + /// See `MipsCache::set_cached_view`. + /// + /// # Safety + /// `sp` must outlive the CPU. + unsafe fn set_cached_view(&self, sp: *const crate::ppmem::PpMemSpace, view: usize); /// tcache: see `MipsExecutor::set_tcache_shift`. fn set_tcache_shift(&self, shift: u32); /// tcache: pointer to the cache's inline mapped-region bitmap. @@ -11367,6 +11388,9 @@ impl CpuDevice for MipsCp unsafe fn set_tcache_window(&self, base: *mut u8) { unsafe { MipsCpu::set_tcache_window(self, base) } } + unsafe fn set_cached_view(&self, sp: *const crate::ppmem::PpMemSpace, view: usize) { + unsafe { MipsCpu::set_cached_view(self, sp, view) } + } fn tcache_bitmap_ptr(&self) -> *mut u64 { MipsCpu::tcache_bitmap_ptr(self) } #[cfg(feature = "jitv2")] unsafe fn set_tcache_gen_window(&self, gen_base: *mut AtomicU64) { diff --git a/src/dev/crime_mem.rs b/src/dev/crime_mem.rs index 4b362456f..f8501c5cd 100644 --- a/src/dev/crime_mem.rs +++ b/src/dev/crime_mem.rs @@ -49,11 +49,10 @@ //! ([`RamWindow`]). Every other master reaches memory through the bus. //! //! POST keeps its stack in cache lines tagged for physical page 0 while it -//! tests the same page uncached, so on the machine the two are different bytes. -//! The CPU's cached path and the window always mean the bank, so that is where -//! the stack lives here too, and for as long as those tags stand the decode -//! table points that page at a page of its own for the uncached path. See -//! [`O2Memory::stack_tag_store`]. +//! tests the same page uncached, so on the machine the two are different +//! bytes. The lines live in the CPU's cached view of the window +//! (`PpMemSpace::add_cached_view`), and the bank, the decode table and every +//! other master go on seeing memory. //! //! `O2Memory` is CRIME's bank logic: the sizes fitted, the controls, the //! decode they make, and the remap. Its decode is atomics: the firmware @@ -84,13 +83,6 @@ const SLOTS: usize = (RAM_WINDOW_SIZE / SLOT) as usize; /// A 32 MB slot in the decode table's 4 KB entries. const TABLE_PER_SLOT: u64 = (SLOT >> SLOT_SHIFT) as u64; -/// The one page of physical memory POST keeps its stack in. -const STACK_PAGE: u32 = 4096; -/// TagLo as R10000 and R5000 share it: the primary state, 0 for invalid. -const TAG_STATE: u32 = 0xc0; -/// TagLo's physical page number. -const TAG_PAGE_SHIFT: u32 = 8; - /// The kits SGI sold, in MB per pair. pub const KITS_MB: [u32; 4] = [32, 64, 128, 256]; @@ -179,12 +171,6 @@ pub struct O2Memory { /// What [`RamWindow`]s need to know of the decode, published as it /// changes. shared: Arc, - /// Page 0 as the uncached path sees it while POST's stack is in the bank. - /// It wraps at 4 KB, so the same device answers at every alias. Boxed: the - /// decode table points at it and `O2Memory` is built by value. - stack_page: Box, - /// Whether the decode table is pointing at [`Self::stack_page`]. - stack_live: AtomicBool, } impl O2Memory { @@ -241,8 +227,6 @@ impl O2Memory { } let phys = Physical::new(banks); phys.set_default(Arc::new(NoMemory), "no memory"); - let stack_page = Box::new(RamBank::new(1)); - stack_page.set_addr_mask(STACK_PAGE - 1); let m = Self { phys, size, @@ -253,8 +237,6 @@ impl O2Memory { decode_gen: AtomicU64::new(0), rebuilding: std::sync::Mutex::new(()), shared: Default::default(), - stack_page, - stack_live: AtomicBool::new(false), }; // At reset every control is zero: all eight banks at the bottom of // the window, 32 MB each, so bank 0 answers there. @@ -356,106 +338,10 @@ impl O2Memory { let _ = sp.map_range(i, src, at, SLOT as u64); } } - if self.stack_live.load(Ordering::Relaxed) { - self.place_stack_page(slots, mapped); - } self.shared.decode_gen.store(self.decode_gen.load(Ordering::Acquire), Release); self.shared.mapped.store(mapped, Release); } - /// The decode table's entry for bank 0's offset 0 goes to the stack page, - /// at every place that offset answers: a 32 MB bank under a 128 MB setting - /// repeats, and sizing relies on it. - fn place_stack_page(&self, slots: &[Slot; SLOTS], mapped: u32) { - let phys = &*self.phys; - let page: *const RamBank = &*self.stack_page; - let page = page as crate::ppmem::devmap::Entry; - let size = self.size[0] as u64; - for (s, slot) in slots.iter().enumerate() { - let Slot::Ram { bank: 0, chunk } = *slot else { continue }; - if (chunk as u64 * SLOT as u64) & (size - 1) != 0 { - continue; - } - let at = s as u64 * SLOT as u64; - phys.fill_slots((RAM_BASE as u64 + at) >> SLOT_SHIFT, 1, page); - if at < LOW_ALIAS_SIZE as u64 { - phys.fill_slots(at >> SLOT_SHIFT, 1, page); - } - } - // ppmem's bitmap wants a whole 64 MB region mapped. At reset only the - // first 32 MB slot is, so the cached path would take the bus and find - // the stack page, then find the bank once sizing filled the region. - // Show bank 0 again in the next slot, for the CPU only. - if mapped & 2 == 0 { - if let (Some(sp), Some(i)) = (phys.ppmem_space(), self.index[0]) { - let _ = sp.map_range(i, 0, SLOT as u64, SLOT as u64); - } - } - } - - /// The guest stored `tag` in a primary data cache tag. - /// - /// POST sweeps a page's worth of tags to dirty for physical page 0 and - /// fills them, with nothing read from memory, then writes the same page - /// uncached to test it, and sweeps every tag invalid without a writeback - /// once it has copied what it needs and moved its stack to KSEG1. So a - /// valid tag for page 0 starts the stack and the next invalid tag ends it. - /// IRIX reboots by jumping into the PROM, so this catches a reboot as it - /// does a power-on. Kernels and `sash` store invalid tags only, which is - /// nothing to do here. - pub fn stack_tag_store(&self, tag: u32) { - if tag & TAG_STATE == 0 { - if self.stack_live.load(Ordering::Relaxed) { - self.set_stack_page(false); - } - } else if tag >> TAG_PAGE_SHIFT == 0 && !self.stack_live.load(Ordering::Relaxed) { - self.set_stack_page(true); - } - } - - /// Start the stack (the uncached path gets a copy of the page, the bank - /// goes on holding what the cached path sees) or end it (what was written - /// uncached is what memory holds, as it is when the cache lines go). - pub fn set_stack_page(&self, on: bool) { - let _one = self.rebuilding.lock().unwrap(); - if self.stack_live.load(Ordering::Relaxed) == on { - return; - } - let bank = &self.phys.banks()[self.index[0].expect("bank 0 is fitted")]; - for o in (0..STACK_PAGE as u64).step_by(8) { - if on { - self.stack_page.write64(o, bank.read64(o).data); - } else { - bank.write64(o, self.stack_page.read64(o).data); - } - } - self.stack_live.store(on, Ordering::Relaxed); - self.rebuild_locked(); - } - - /// The stack page's dwords while the stack is live, for a snapshot: the - /// bank is saved with memory and this is what the uncached path sees. - pub fn stack_page_words(&self) -> Option> { - self.stack_live.load(Ordering::Relaxed).then(|| { - (0..STACK_PAGE as u64).step_by(8).map(|o| self.stack_page.read64(o).data).collect() - }) - } - - /// Put back what [`Self::stack_page_words`] gave. The stack is not live - /// beforehand (a machine being restored has been powered on). - pub fn restore_stack_page(&self, words: &[u64]) -> Result<(), String> { - if words.len() * 8 != STACK_PAGE as usize { - return Err(format!("saved stack page is {} dwords", words.len())); - } - let _one = self.rebuilding.lock().unwrap(); - for (k, &w) in words.iter().enumerate() { - self.stack_page.write64(k as u64 * 8, w); - } - self.stack_live.store(true, Ordering::Relaxed); - self.rebuild_locked(); - Ok(()) - } - /// A view of memory through the window, for a master on its own thread /// (the rendering engine). pub fn window(&self) -> RamWindow { @@ -1118,92 +1004,6 @@ mod tests { assert_eq!(w, 0xbeef, "the window agrees"); } - const STACK_TAG: u32 = 0xc8; // dirty, page 0: what POST stores on an R10000 - - /// What each path sees of page 0 once the stack is live, and that memory - /// ends up holding what was written uncached. - #[test] - fn the_stack_and_the_uncached_page_are_different_bytes() { - let m = O2Memory::with_total(64).unwrap(); - m.set_ctrl(0, 0); - let bus = m.phys(); - let base = bus.ppmem_space().unwrap().window_base(); - let cached = |o: usize, base: *mut u8| unsafe { (*(base.add(o) as *const u64)).rotate_left(32) }; - let store = |o: usize, v: u64, base: *mut u8| unsafe { *(base.add(o) as *mut u64) = v.rotate_left(32) }; - assert_eq!(bus.write64(0xff0, 0x1111), BUS_OK); - m.stack_tag_store(0x0); - m.stack_tag_store(STACK_TAG); - assert_eq!(bus.read64(0xff0).data, 0x1111, "the uncached page starts as a copy"); - store(0xff0, 0x2222, base); - assert_eq!(bus.write64(0xff8, 0x3333), BUS_OK); - assert_eq!(cached(0xff0, base), 0x2222, "the stack, cached"); - assert_eq!(cached(0xff8, base), 0, "the test's write does not reach the stack"); - assert_eq!(bus.read64(0xff0).data, 0x1111, "and the stack does not reach the test"); - assert_eq!(bus.read64(0xff8).data, 0x3333); - assert_eq!(bus.read64(RAM_BASE as u64 + 0xff8).data, 0x3333, "through the window's address too"); - assert_eq!(bus.write64(0x1000, 0x4444), BUS_OK); - assert_eq!(cached(0x1000, base), 0x4444, "the next page is memory"); - m.stack_tag_store(0x0); - assert_eq!(cached(0xff8, base), 0x3333, "memory holds what was written uncached"); - assert_eq!(bus.read64(0xff0).data, 0x1111); - store(0xff8, 0x5555, base); - assert_eq!(bus.read64(0xff8).data, 0x5555, "one memory again"); - } - - /// Sizing reads back through a 32 MB bank repeating under a 128 MB - /// setting, so page 0 has to answer the same way at every repeat. - #[test] - fn the_uncached_page_follows_a_small_bank_repeating() { - let m = O2Memory::with_total(32).unwrap(); - m.stack_tag_store(STACK_TAG); - m.set_ctrl(0, CTRL_128M); - put(&m, RAM_BASE + 8, 0xfeed); - assert_eq!(get(&m, RAM_BASE + 3 * SLOT + 8), Some(0xfeed)); - assert_eq!(get(&m, 3 * SLOT + 8), Some(0xfeed), "the low alias"); - m.stack_tag_store(0); - assert_eq!(m.phys().banks()[0].read64(8).data, 0xfeed); - } - - /// At reset one 32 MB slot is mapped, which ppmem's bitmap does not count; - /// the cached path must still reach the bank, not the bus, or the stack - /// moves out from under POST when sizing completes the region. - #[test] - fn the_cached_path_is_on_the_bank_from_reset() { - let m = O2Memory::with_total(32).unwrap(); - let sp = m.phys().ppmem_space().unwrap(); - let region = 1u64 << 0; - assert_eq!(sp.mapped_bitmap() & region, 0, "a hole at reset"); - m.stack_tag_store(STACK_TAG); - assert_ne!(sp.mapped_bitmap() & region, 0, "filled while the stack is live"); - m.stack_tag_store(0); - assert_eq!(sp.mapped_bitmap() & region, 0, "and only then"); - } - - /// Only a valid tag for page 0 starts the stack; other firmware's tags - /// and a kernel's invalid ones leave the decode alone. - #[test] - fn other_tags_leave_the_decode_alone() { - let m = O2Memory::with_total(32).unwrap(); - for tag in [0u32, 0x1_00c8, 0x08] { - m.stack_tag_store(tag); - assert!(m.stack_page_words().is_none(), "{tag:#x}"); - } - } - - /// A snapshot taken during POST comes back with both views. - #[test] - fn a_snapshot_carries_the_uncached_page() { - let m = O2Memory::with_total(32).unwrap(); - m.stack_tag_store(STACK_TAG); - put(&m, 0x20, 0xabcd); - let words = m.stack_page_words().unwrap(); - let n = O2Memory::with_total(32).unwrap(); - n.restore_stack_page(&words).unwrap(); - assert_eq!(get(&n, 0x20), Some(0xabcd)); - n.stack_tag_store(0); - assert_eq!(n.phys().banks()[0].read64(0x20).data, 0xabcd); - } - /// An empty bank's slots read zero; with no control matching, the bus's /// default answers. #[test] diff --git a/src/ip27_machine.rs b/src/ip27_machine.rs index 73bfd0a98..de91a65cd 100644 --- a/src/ip27_machine.rs +++ b/src/ip27_machine.rs @@ -497,21 +497,21 @@ impl Ip27Machine { } else { crate::with_cpu!(module.part, true, false, C => Self::cpu::(module, bus, phys, memory, &node.timers, count_hz, cpu_id, smp)) }; - // The shadow cache and the JIT read RAM straight from the ppmem window. + // The shadow cache and the JIT read RAM through a cached view of the + // ppmem window that is the CPU's own. if let Some(sp) = node.memory.physical().ppmem_space() { - use crate::ppmem::MappedMemory; + let v = sp + .add_cached_view(&[cpu.ppmem_bitmap_ptr(), cpu.tcache_bitmap_ptr()]) + .map_err(|e| format!("ppmem: a CPU's cached view: {e}"))?; unsafe { - sp.set_bitmap_sink(cpu.ppmem_bitmap_ptr()); - cpu.set_tcache_window(sp.window_base()); + cpu.set_tcache_window(sp.view_base(v)); cpu.set_tcache_shift(crate::ppmem::window_shift(sp.window_size())); - let bitmap = cpu.tcache_bitmap_ptr(); - if !bitmap.is_null() { - sp.set_bitmap_sink2(bitmap); - } #[cfg(feature = "jitv2")] - cpu.set_tcache_gen_window(sp.gen_window_base()); + { + use crate::ppmem::MappedMemory; + cpu.set_tcache_gen_window(sp.gen_window_base()); + } } - node.memory.physical().resync_ppmem_bitmap(); } #[cfg(feature = "jitv2")] { diff --git a/src/ip30_machine.rs b/src/ip30_machine.rs index 7b434e1f9..7ae210e32 100644 --- a/src/ip30_machine.rs +++ b/src/ip30_machine.rs @@ -395,33 +395,25 @@ impl OctaneMachine { iris_hostcall::set_display(Arc::new(crate::dev::mgras::ImpactScreen(g.mgras().clone()))); } - // The shadow cache and the JIT read RAM straight from the ppmem - // window; descriptor writes lay it out again on the CPU thread. + // The shadow cache and the JIT read RAM through a cached view of the + // ppmem window, one per CPU; descriptor writes lay the window and + // every view out again on the CPU thread. if let Some(sp) = phys.phys.ppmem_space() { - use crate::ppmem::MappedMemory; - unsafe { - sp.set_bitmap_sink(cpu.ppmem_bitmap_ptr()); - cpu.set_tcache_window(sp.window_base()); - cpu.set_tcache_shift(crate::ppmem::window_shift(sp.window_size())); - let bitmap = cpu.tcache_bitmap_ptr(); - if !bitmap.is_null() { - sp.set_bitmap_sink2(bitmap); - } - #[cfg(feature = "jitv2")] - cpu.set_tcache_gen_window(sp.gen_window_base()); - for c in cpus.iter().skip(1) { - sp.add_bitmap_sink(c.ppmem_bitmap_ptr()); - c.set_tcache_window(sp.window_base()); + for c in cpus.iter() { + let v = sp + .add_cached_view(&[c.ppmem_bitmap_ptr(), c.tcache_bitmap_ptr()]) + .map_err(|e| format!("ppmem: a CPU's cached view: {e}"))?; + unsafe { + c.set_tcache_window(sp.view_base(v)); + c.set_cached_view(sp, v); c.set_tcache_shift(crate::ppmem::window_shift(sp.window_size())); - let bitmap = c.tcache_bitmap_ptr(); - if !bitmap.is_null() { - sp.add_bitmap_sink(bitmap); - } #[cfg(feature = "jitv2")] - c.set_tcache_gen_window(sp.gen_window_base()); + { + use crate::ppmem::MappedMemory; + c.set_tcache_gen_window(sp.gen_window_base()); + } } } - phys.phys.resync_ppmem_bitmap(); } // Publish the descriptors as they stand into the sinks just set. ram.set_memcfg(heart.memcfg()); diff --git a/src/ip32.rs b/src/ip32.rs index 1069ae454..27f220cb8 100644 --- a/src/ip32.rs +++ b/src/ip32.rs @@ -667,14 +667,18 @@ impl crate::traits::Saveable for Ip32Bus { t.insert("mace".into(), self.mace.save_state()); let scsi = self.scsi.lock().unwrap().iter().map(|c| c.save_state()).collect(); t.insert("scsi".into(), toml::Value::Array(scsi)); - if let Some(words) = self.ram.stack_page_words() { - t.insert("stack_page".into(), toml::Value::Array(words.into_iter().map(crate::snapshot::hex_u64).collect())); - } toml::Value::Table(t) } fn load_state(&self, v: &toml::Value) -> Result<(), String> { use crate::snapshot::get_field; + // Older snapshots taken while POST's stack stood kept the page the + // uncached path saw. The stack lives in the CPU's cached view now, + // and they hold no cache lines to rebuild it from. + if get_field(v, "stack_page").is_some() { + return Err("this snapshot was taken during POST by an older iris, which kept the \ + stack page outside the cache; take it again".into()); + } if let Some(m) = get_field(v, "mace") { self.mace.load_state(m)?; } @@ -696,10 +700,6 @@ impl crate::traits::Saveable for Ip32Bus { c.load_state(s)?; } } - if let Some(toml::Value::Array(a)) = get_field(v, "stack_page") { - let words: Option> = a.iter().map(crate::snapshot::toml_u64).collect(); - self.ram.restore_stack_page(&words.ok_or("stack_page: not a list of dwords")?)?; - } Ok(()) } } @@ -710,7 +710,6 @@ impl Ip32Bus { /// on the hardware. The PROM sets up the rest again as it starts. pub fn power_on(&self) { use crate::traits::Resettable; - self.ram.set_stack_page(false); self.crime.regs.power_on(); let m = &self.mace; let devices: [&dyn Resettable; 8] = diff --git a/src/ip32/tests.rs b/src/ip32/tests.rs index 7031e5185..8f19dfd25 100644 --- a/src/ip32/tests.rs +++ b/src/ip32/tests.rs @@ -214,6 +214,18 @@ fn a_bus_saves_and_loads_whole() { std::fs::remove_dir_all(&dir).ok(); } +/// A snapshot an older iris took during POST kept the stack page outside +/// the cache, which nothing can put back now: it is refused. +#[test] +fn a_snapshot_with_the_old_stack_page_is_refused() { + use crate::traits::Saveable; + let bus = Ip32Bus::new(16 << 20, vec![0u8; PROM_SIZE as usize]); + let mut v = bus.save_state(); + v.as_table_mut().unwrap().insert("stack_page".into(), toml::Value::Array(vec![])); + let e = bus.load_state(&v).unwrap_err(); + assert!(e.contains("take it again"), "{e}"); +} + /// Setting CONTROL's cold-reset bit asks the machine for a reset once, and /// the bit reads back clear, as the hardware clears it. #[test] diff --git a/src/ip32_machine.rs b/src/ip32_machine.rs index 1eb5c95b2..98fa4445e 100644 --- a/src/ip32_machine.rs +++ b/src/ip32_machine.rs @@ -240,27 +240,26 @@ impl O2Machine { let boot = Arc::new(BootState::new()); let memory = Arc::new(crate::ip32::memory_map()); let cpu: Arc = crate::with_cpu!(part, true, true, C => - Self::build_cpu::(memory.clone(), sysad, &timer_manager, arcs_fpu, count_hz, boot.clone(), bus.ram.clone(), part.prid())); + Self::build_cpu::(memory.clone(), sysad, &timer_manager, arcs_fpu, count_hz, boot.clone(), part.prid())); - // ppmem: the CPU's cache and the JIT reach RAM through the memory's - // window, publishing the mapped bitmap into the CPU's own fields, as - // the Indigo2 does. Remaps happen on the CPU thread (a CRIME bank - // control write), which is the window's contract. + // ppmem: the CPU's cache and the JIT reach RAM through a cached view + // of the memory's window, publishing its bitmap into the CPU's own + // fields, as the Indigo2 does. Remaps happen on the CPU thread (a + // CRIME bank control write), which is the window's contract. { - use crate::ppmem::MappedMemory; let sp = bus.phys.ppmem_space().ok_or("ppmem: the O2 needs its memory window")?; + let v = sp + .add_cached_view(&[cpu.ppmem_bitmap_ptr(), cpu.tcache_bitmap_ptr()]) + .map_err(|e| format!("ppmem: the CPU's cached view: {e}"))?; unsafe { - sp.set_bitmap_sink(cpu.ppmem_bitmap_ptr()); - cpu.set_tcache_window(sp.window_base()); - let bitmap = cpu.tcache_bitmap_ptr(); - if !bitmap.is_null() { - sp.set_bitmap_sink2(bitmap); - } + cpu.set_tcache_window(sp.view_base(v)); + cpu.set_cached_view(sp, v); #[cfg(feature = "jitv2")] - cpu.set_tcache_gen_window(sp.gen_window_base()); + { + use crate::ppmem::MappedMemory; + cpu.set_tcache_gen_window(sp.gen_window_base()); + } } - // The bus's direct paths read the same bitmap. - bus.phys.resync_ppmem_bitmap(); } // After the windows above: each compile worker takes the cache's @@ -361,7 +360,7 @@ impl O2Machine { } fn build_cpu(memory: Arc, sysad: Arc, tm: &Arc, arcs_fpu: bool, count_hz: u64, boot: Arc, - ram: Arc, prid: u32) -> Arc + prid: u32) -> Arc where C: crate::cpu::mips_cache_v2::CpuModel + MipsCache + From> + 'static, MipsCpu: CpuDevice, @@ -369,7 +368,6 @@ impl O2Machine { let cfg = MipsCpuConfig::for_model::(memory).with_prid(prid); let tlb = MipsTlb::new(cfg.tlb_entries); let mut exec: MipsExecutor = MipsExecutor::new(sysad, tlb, &cfg); - exec.cache.set_pd_tag_store_hook(Box::new(move |tag| ram.stack_tag_store(tag))); exec.core.set_timer_manager(tm.clone()); // Count runs at half the pipeline clock on both CPUs. exec.core.set_count_hz(count_hz.max(1)); diff --git a/src/machine.rs b/src/machine.rs index 9efbfe6e6..62566f107 100644 --- a/src/machine.rs +++ b/src/machine.rs @@ -800,33 +800,22 @@ impl Machine { mc.set_cpu(Arc::downgrade(&cpu_device)); ioc.set_interrupts(cpu.interrupts_ptr()); - // ppmem: hand the CPU's inline bitmap field to the memory space, so a - // MEMCFG remap publishes straight into the object the CPU executes out - // of. Safe per `set_bitmap_sink`: the executor is at its final address - // by now and the only writer is this same CPU thread's MEMCFG path. + // ppmem: the CPU's cache and the JIT read RAM through a cached view of + // the window that is the CPU's own, so what the cache holds stays out + // of every other master's sight (`PpMemSpace::add_cached_view`). Its + // bitmap is published straight into the CPU's inline fields, the + // executor's and the cache's, which are at their final addresses by + // now; the only writer is this CPU thread's MEMCFG path. A cache with + // no bitmap field (the full cache without tcache) returns null, which + // the view skips. Physical's direct paths keep the shared window. { use crate::ppmem::MappedMemory; if let Some(sp) = phys.ppmem_space() { - unsafe { sp.set_bitmap_sink(cpu.ppmem_bitmap_ptr()) }; - } - // Point Physical's cached bitmap at the CPU's field too, or its - // direct-access fast path would keep reading the sink the space - // just abandoned. - phys.resync_ppmem_bitmap(); - // The cache reads RAM straight out of the same window: the full - // cache under tcache, the shadow cache always. A cache that does - // not (the full cache without tcache) has no bitmap field and - // returns null. Must come after set_bitmap_sink, so it captures - // the CPU's inline field rather than the sink the space just - // abandoned. - if let Some(sp) = phys.ppmem_space() { - unsafe { cpu.set_tcache_window(sp.window_base()) }; - // The cache keeps the bitmap inline; ppmem publishes into it - // directly on every remap, so the hot path needs no indirection. - let bitmap = cpu.tcache_bitmap_ptr(); - if !bitmap.is_null() { - unsafe { sp.set_bitmap_sink2(bitmap) }; - } + let v = sp + .add_cached_view(&[cpu.ppmem_bitmap_ptr(), cpu.tcache_bitmap_ptr()]) + .expect("ppmem: the CPU's cached view"); + unsafe { cpu.set_tcache_window(sp.view_base(v)) }; + unsafe { cpu.set_cached_view(sp, v) }; // Window writes bypass BusDevice, so the cache needs the gen // window to bump page counters itself — otherwise jitv2 keeps // running compiled code for pages the guest has modified. diff --git a/src/ppmem/ppmem.rs b/src/ppmem/ppmem.rs index a30c0adb5..e6da8ba9e 100644 --- a/src/ppmem/ppmem.rs +++ b/src/ppmem/ppmem.rs @@ -151,32 +151,14 @@ pub trait MappedMemory { /// a finer, multi-word bitmap. fn mapped_bitmap(&self) -> u64; - /// Redirect bitmap publication into the CPU's inline `ppmem_bitmap` field. - /// - /// Pass `MipsExecutor::ppmem_bitmap_ptr()`, queried once the executor is at - /// its final address. From then on every mapping change writes straight - /// through this pointer, so the CPU's RAM-vs-bus test is a load from the - /// object it is already executing out of — no indirection through - /// `PpMemSpace`, no atomic, no lock. - /// - /// Optional: until this is called the space publishes into a `u64` it owns - /// itself, so the sink is always valid and publication never branches. - /// - /// # Safety - /// - /// `ptr` must point to a live `u64` that outlives this `PpMemSpace`, and - /// must only ever be written from the CPU thread. Both hold for the CPU's - /// inline field: the executor never moves once inside its `Arc`, and the - /// only writer is the MEMCFG path on that same thread. - unsafe fn set_bitmap_sink(&self, ptr: *mut u64); - - /// A second publication target — see [`MappedMemory::set_bitmap_sink`]. + /// A second publication target, the cache's inline copy. /// /// Exists so the cache can hold the bitmap *inline* instead of chasing a /// pointer on every access. Both sinks receive every update. /// /// # Safety - /// Same contract as `set_bitmap_sink`. + /// `ptr` must point to a live `u64` that outlives this `PpMemSpace` and is + /// written only from the CPU thread. unsafe fn set_bitmap_sink2(&self, ptr: *mut u64); } @@ -656,11 +638,17 @@ impl BusDevice for Arc { // --------------------------------------------------------------------------- /// One live mapping inside the window, tracked so `clear_mappings` can undo it -/// and `mapped_bitmap` can be recomputed. +/// and `mapped_bitmap` can be recomputed. It also says where its bytes come +/// from, so a cached view can lay part of it again when a held page leaves: +/// `bank`'s object from `src`, or repeating every `period` bytes from `src` +/// into the period (0: no repeat). #[derive(Clone, Copy, Debug)] struct Mapping { at: u64, len: u64, + bank: usize, + src: u64, + period: u64, } /// The 4GB window that banks are mapped into, plus the banks themselves. @@ -692,7 +680,7 @@ pub struct PpMemSpace { /// `mapped_bitmap()`; written only via `state.sink`. own_bitmap: OwnBitmap, /// No lock: every path that touches this — `map_bank`, `map_alias`, - /// `clear_mappings`, `set_bitmap_sink` — runs on the CPU thread, reached + /// `clear_mappings`, `set_bitmap_sink2` — runs on the CPU thread, reached /// as CPU store → MC register write → `remap_banks`. No device thread ever /// enters this type. (`unsafe impl Sync` below records that contract.) state: UnsafeCell, @@ -711,8 +699,23 @@ struct SpaceState { /// path is one load instead of two — hence one sink per consumer rather /// than a shared indirection. Points at `own_bitmap` until claimed. sink2: *mut u64, - /// Further CPUs' inline copies (`PpMemSpace::add_bitmap_sink`). - more_sinks: Vec<*mut u64>, + /// The bitmap last published to the sinks above. + bits: u64, + /// Each CPU's cached view of memory: see [`PpMemSpace::add_cached_view`]. + views: Vec, +} + +/// One CPU's cached view of memory: a window of its own, with every mapping +/// the shared window has, plus the pages its cache holds lines for, which +/// nobody else sees. +struct View { + space: AddrSpace, + /// Where this view's region bitmap is published: the CPU's inline copies. + sinks: Vec<*mut u64>, + /// The pages this view holds, with how many of the CPU's valid lines each + /// holds. Only its CPU changes them, at CACHE instructions, under the + /// contract below. + held: std::collections::BTreeMap, } /// The bitmap `PpMemSpace` publishes into before a CPU claims it. @@ -727,12 +730,15 @@ struct OwnBitmap(Box); // formality: // // * Every mutating entry point — `map_bank`, `map_alias`, `clear_mappings`, -// `set_bitmap_sink` — is reached only from `Physical::remap_banks`, which +// `set_bitmap_sink2` — is reached only from `Physical::remap_banks`, which // the MC invokes synchronously inside the CPU's store to MEMCFG. That is // always a CPU thread, and with several CPUs the others are parked for it // (`smp::World::stop_others`, wired where HEART's descriptors reach memory). // * Device threads (MC-DMA, seeq, VINO) reach RAM through `BusDevice` on the // individual `PpMemory` banks. They never touch `PpMemSpace`. +// * `hold_line_page`/`release_line_page` run on a CPU thread at a CACHE +// instruction and change only that CPU's own view. They read the mapping +// list and the bitmap, which only a remap changes, with the others parked. // * `mapped_bitmap()` reads the sink and is likewise CPU-thread-only; the // hot-path reader is the CPU's own inline `MipsCore::ppmem_bitmap` field, // which is a plain `u64` and never goes through here. @@ -769,7 +775,8 @@ impl PpMemSpace { mappings: Vec::new(), sink, sink2, - more_sinks: Vec::new(), + bits: 0, + views: Vec::new(), }), }) } @@ -792,8 +799,8 @@ impl PpMemSpace { /// The `u64` the mapped-region bitmap is published into. /// - /// Never null: points at this space's own `u64` until a CPU claims it via - /// [`MappedMemory::set_bitmap_sink`], and at the CPU's inline field after. + /// Never null: this space's own `u64`. A CPU reads its cached view's + /// copy instead (`add_cached_view`). pub fn bitmap_ptr(&self) -> *const u64 { let st = unsafe { &*self.state.get() }; st.sink as *const u64 @@ -811,9 +818,11 @@ impl PpMemSpace { let b = &self.banks[bank]; assert!(src + size <= b.len() as u64, "ppmem: range [{src:#x}, +{size:#x}) escapes bank {bank}"); assert!(offset + size <= self.window_size(), "ppmem: [{offset:#x}, +{size:#x}) escapes the window"); + self.marks(false); unsafe { self.space.map(offset as usize, size as usize, b, src, Prot::ReadWrite)?; } + self.map_views(offset, size, b, src)?; #[cfg(feature = "jitv2")] { let (gen_off, gen_len, gen_src) = (offset / GEN_RATIO, size / GEN_RATIO, src / GEN_RATIO); @@ -827,7 +836,8 @@ impl PpMemSpace { } } let st = unsafe { &mut *self.state.get() }; - Self::record(st, offset, size); + Self::record(st, Mapping { at: offset, len: size, bank, src, period: 0 }); + self.marks(true); Ok(()) } @@ -867,37 +877,273 @@ impl PpMemSpace { unsafe { *state.sink = bits; *state.sink2 = bits; - for &s in &state.more_sinks { - *s = bits; - } + } + state.bits = bits; + for v in 0..state.views.len() { + Self::publish_view(state, v); } } - fn record(state: &mut SpaceState, at: u64, len: u64) { + /// Publish view `v`'s bitmap: the shared one, plus every region the view + /// holds a page in, which `hold_line_page` made safe to touch whole. + fn publish_view(state: &SpaceState, v: usize) { + let shift = window_shift(state.window); + let view = &state.views[v]; + let bits = view.held.keys().fold(state.bits, |b, &page| b | 1u64 << (page >> shift)); + for &s in &view.sinks { + unsafe { *s = bits }; + } + } + + fn record(state: &mut SpaceState, m: Mapping) { // Drop anything this mapping fully replaces, then record it. + let (at, len) = (m.at, m.len); state .mappings .retain(|m| !(m.at >= at && m.at + m.len <= at + len)); - state.mappings.push(Mapping { at, len }); + state.mappings.push(m); Self::recompute_bitmap(state); } } +/// Cached views. +/// +/// A CPU's cache and its compiled code read memory through ppmem's window, so +/// the window is that CPU's cached view of memory. Giving each CPU a window of +/// its own makes room for what the cache holds: a valid cache line's page can +/// become private memory in that CPU's view while the shared window, the +/// decode table and every other CPU go on seeing memory. Uncached accesses +/// bypass the caches (NEC VR10000 user's manual p. 75), and each CPU's cache +/// is its own. The views are not coherent: on a coherent system another +/// master's read would be answered from a dirty line, and here it reads +/// memory. +/// +/// Nothing on a load, store or fetch changes: the CPU reads its view through +/// the same base pointer and inline bitmap it read the shared window through. +/// The page is the host's mapping granule (4 KB on Linux and x86 macOS, +/// 16 KB on Apple silicon, 64 KB on Windows). impl PpMemSpace { - /// Publish the bitmap to one more `u64` as well: a second CPU's inline - /// copies. Same contract as [`MappedMemory::set_bitmap_sink`]. The value - /// arrives with the next remap. + /// Give a CPU a cached view of its own: a window with the same mappings + /// as this one, from now on, whose region bitmap is published to `sinks` + /// (the CPU's inline copies; null entries are skipped) instead of this + /// window's. Returns the view's number. Set up before the CPU runs. + pub fn add_cached_view(&self, sinks: &[*mut u64]) -> io::Result { + let st = unsafe { &mut *self.state.get() }; + let space = AddrSpace::reserve(st.window as usize)?; + let sinks = sinks.iter().copied().filter(|p| !p.is_null()).collect(); + st.views.push(View { space, sinks, held: Default::default() }); + let v = st.views.len() - 1; + for m in st.mappings.clone() { + self.lay(st, v, &m, m.at, m.at + m.len)?; + } + Self::publish_view(st, v); + Ok(v) + } + + /// The base of view `v`'s window. + pub fn view_base(&self, v: usize) -> *mut u8 { + unsafe { (&(*self.state.get()).views)[v].space.base() } + } + + /// The pages view `v` holds, in address order. + pub fn held_line_pages(&self, v: usize) -> Vec { + unsafe { (&(*self.state.get()).views)[v].held.keys().copied().collect() } + } + + /// A line of view `v`'s CPU at `pa` became valid: from now on the view + /// shows the page holding it as private memory, starting as a copy of + /// memory. Each call is one line, and [`Self::release_line_page`] gives + /// it back. Returns whether the page is newly held, or `None` when no + /// memory is mapped behind it. /// - /// # Safety - /// `ptr` must stay valid for as long as this space exists. - pub unsafe fn add_bitmap_sink(&self, ptr: *mut u64) { - unsafe { (*self.state.get()).more_sinks.push(ptr) }; + /// The region bitmap is all a CPU tests, so the page's whole region is + /// the view's while it stands, reading zeros where nothing is mapped. The + /// O2's POST holds lines for page 0 before CRIME maps the rest of its + /// region. + pub fn hold_line_page(&self, v: usize, pa: u64) -> Option { + let st = unsafe { &mut *self.state.get() }; + let gran = granularity() as u64; + let page = pa & !(gran - 1); + if !st.mappings.iter().any(|m| m.at <= page && m.at + m.len >= page + gran) { + return None; + } + let shift = window_shift(st.window); + if let Some(n) = st.views[v].held.get_mut(&page) { + *n += 1; + return Some(false); + } + if !bitmap_maps_at(st.bits, shift, page) && !st.views[v].held.keys().any(|&p| p >> shift == page >> shift) { + self.scrub_region_gaps(st, v, page >> shift << shift, 1u64 << shift); + } + if !Self::held_anywhere(st, page) { + self.mark(st, page, true); + } + let view = &st.views[v]; + let at = unsafe { view.space.base().add(page as usize) }; + let mut bytes = vec![0u8; gran as usize]; + unsafe { + std::ptr::copy_nonoverlapping(at, bytes.as_mut_ptr(), bytes.len()); + let _ = view.space.scrub(page as usize, gran as usize); + std::ptr::copy_nonoverlapping(bytes.as_ptr(), at, bytes.len()); + } + st.views[v].held.insert(page, 1); + Self::publish_view(st, v); + Some(true) + } + + /// One line of [`Self::hold_line_page`]'s page is no longer valid. + /// Returns true when it was the page's last: the view shows memory there + /// again, and what the page held is dropped, as a line invalidated + /// without writeback is. + pub fn release_line_page(&self, v: usize, pa: u64) -> bool { + let st = unsafe { &mut *self.state.get() }; + let gran = granularity() as u64; + let page = pa & !(gran - 1); + let Some(n) = st.views[v].held.get_mut(&page) else { return false }; + *n -= 1; + if *n > 0 { + return false; + } + st.views[v].held.remove(&page); + if !Self::held_anywhere(st, page) { + self.mark(st, page, false); + } + let _ = unsafe { st.views[v].space.scrub(page as usize, gran as usize) }; + for m in st.mappings.iter().filter(|m| m.at < page + gran && m.at + m.len > page) { + let _ = self.lay(st, v, m, page.max(m.at), (page + gran).min(m.at + m.len)); + } + Self::publish_view(st, v); + true + } + + fn held_anywhere(st: &SpaceState, page: u64) -> bool { + st.views.iter().any(|view| view.held.contains_key(&page)) + } + + /// A page some view holds carries one `GEN_HELD` in its generation + /// counters while memory backs it, so jitv2 does not compile it. + #[cfg_attr(not(feature = "jitv2"), allow(unused_variables))] + fn mark(&self, st: &SpaceState, page: u64, on: bool) { + #[cfg(feature = "jitv2")] + { + use crate::cpu::jitv2::jitv2::{GEN_HELD, PAGE_SIZE}; + let end = page + granularity() as u64; + if !st.mappings.iter().any(|m| m.at <= page && m.at + m.len >= end) { + return; + } + let d = if on { GEN_HELD } else { 1u64.wrapping_sub(GEN_HELD) }; + let gens = self.gen_space.base() as *const AtomicU64; + for a in (page..end).step_by(PAGE_SIZE as usize) { + unsafe { (*gens.add((a / PAGE_SIZE as u64) as usize)).fetch_add(d, Ordering::Release) }; + } + } + } + + /// Every held page's mark, off before a mapping change and on after. + fn marks(&self, on: bool) { + let st = unsafe { &*self.state.get() }; + let pages: std::collections::BTreeSet = st.views.iter().flat_map(|v| v.held.keys().copied()).collect(); + for page in pages { + self.mark(st, page, on); + } + } + + /// Make the parts of `[r, r + len)` that nothing maps private zeros in + /// view `v`, and give them generation counters where none are mapped, so + /// the region can be marked whole in the view's bitmap: compiled stores + /// there bump a counter. + fn scrub_region_gaps(&self, st: &SpaceState, v: usize, r: u64, len: u64) { + let end = r + len; + let mut at = r; + while at < end { + if let Some(m) = st.mappings.iter().find(|m| m.at <= at && m.at + m.len > at) { + at = m.at + m.len; + continue; + } + let next = st.mappings.iter().filter(|m| m.at > at && m.at < end).map(|m| m.at).min().unwrap_or(end); + let _ = unsafe { st.views[v].space.scrub(at as usize, (next - at) as usize) }; + #[cfg(feature = "jitv2")] + { + let g = granularity() as u64; + let (gs, ge) = ((at / GEN_RATIO) & !(g - 1), (next / GEN_RATIO).div_ceil(g) * g); + let touches = |m: &Mapping| m.at < ge * GEN_RATIO && m.at + m.len > gs * GEN_RATIO; + if !st.mappings.iter().any(touches) { + let _ = unsafe { self.gen_space.scrub(gs as usize, (ge - gs) as usize) }; + } + } + at = next; + } + } + + /// A mapping just laid in the shared window goes into every view too, + /// under the pages each holds. + fn map_views(&self, at: u64, len: u64, b: &SharedMem, src: u64) -> io::Result<()> { + let st = unsafe { &*self.state.get() }; + for view in &st.views { + let saved = Self::save_held(view, at, len); + unsafe { view.space.map(at as usize, len as usize, b, src, Prot::ReadWrite)? }; + Self::put_held(view, saved); + } + Ok(()) + } + + /// Lay `[from, to)` of mapping `m` in view `v`. Data only: the views + /// share the window's generation counters. + fn lay(&self, st: &SpaceState, v: usize, m: &Mapping, from: u64, to: u64) -> io::Result<()> { + let b = &self.banks[m.bank]; + let mut at = from; + while at < to { + let rel = at - m.at; + let (src, run) = if m.period == 0 { + (m.src + rel, to - at) + } else { + let o = (m.src + rel) % m.period; + (o, (m.period - o).min(to - at)) + }; + unsafe { st.views[v].space.map(at as usize, run as usize, b, src, Prot::ReadWrite)? }; + at += run; + } + Ok(()) + } + + /// The bytes of each page `view` holds inside `[at, at + len)`, before + /// a mapping change replaces them. + fn save_held(view: &View, at: u64, len: u64) -> Vec<(u64, Vec)> { + let gran = granularity(); + let base = view.space.base(); + view.held.keys() + .filter(|&&p| p < at + len && p + gran as u64 > at) + .map(|&p| { + let mut b = vec![0u8; gran]; + unsafe { std::ptr::copy_nonoverlapping(base.add(p as usize), b.as_mut_ptr(), gran) }; + (p, b) + }) + .collect() + } + + /// Put [`Self::save_held`]'s pages back, private again. + fn put_held(view: &View, saved: Vec<(u64, Vec)>) { + let base = view.space.base(); + for (page, bytes) in saved { + let _ = unsafe { view.space.scrub(page as usize, bytes.len()) }; + unsafe { std::ptr::copy_nonoverlapping(bytes.as_ptr(), base.add(page as usize), bytes.len()) }; + } } } impl MappedMemory for PpMemSpace { fn clear_mappings(&self) { + self.marks(false); let st = unsafe { &mut *self.state.get() }; + // Views keep the pages they hold. Scrubbed, not left mapped, so a + // view never shows memory the shared window no longer maps. + for view in &st.views { + let saved = Self::save_held(view, 0, st.window); + for m in &st.mappings { + let _ = unsafe { view.space.scrub(m.at as usize, m.len as usize) }; + } + Self::put_held(view, saved); + } for m in std::mem::take(&mut st.mappings) { // Best-effort: a failure here leaves the range mapped, which is // still safe — it just isn't reverted. @@ -915,16 +1161,11 @@ impl MappedMemory for PpMemSpace { } } } - unsafe { - *st.sink = 0; - *st.sink2 = 0; - for &s in &st.more_sinks { - *s = 0; - } - } + Self::recompute_bitmap(st); } fn map_bank(&self, bank: usize, offset: u64, size: u64, period: u64) -> io::Result<()> { + self.marks(false); let b = &self.banks[bank]; let bank_size = b.len() as u64; assert!(size > 0, "ppmem: map_bank with zero size"); @@ -950,6 +1191,7 @@ impl MappedMemory for PpMemSpace { self.space .map(at as usize, chunk as usize, b, 0, Prot::ReadWrite)?; } + self.map_views(at, chunk, b, 0)?; // Lockstep: the counters for this mirror land at the same offset // scaled by GEN_RATIO, from the same bank's gen object. Every // mirror therefore shares one physical counter page with the @@ -975,11 +1217,13 @@ impl MappedMemory for PpMemSpace { } let st = unsafe { &mut *self.state.get() }; - Self::record(st, offset, size); + Self::record(st, Mapping { at: offset, len: size, bank, src: 0, period }); + self.marks(true); Ok(()) } fn map_alias(&self, bank: usize, offset: u64, size: u64) -> io::Result<()> { + self.marks(false); let b = &self.banks[bank]; assert!( size <= b.len() as u64, @@ -994,6 +1238,7 @@ impl MappedMemory for PpMemSpace { self.space .map(offset as usize, size as usize, b, 0, Prot::ReadWrite)?; } + self.map_views(offset, size, b, 0)?; // Alias the counters too — this is NOT optional. // // An earlier version skipped the gen mapping whenever the alias's gen @@ -1045,7 +1290,8 @@ impl MappedMemory for PpMemSpace { } } let st = unsafe { &mut *self.state.get() }; - Self::record(st, offset, size); + Self::record(st, Mapping { at: offset, len: size, bank, src: 0, period: 0 }); + self.marks(true); Ok(()) } @@ -1058,15 +1304,6 @@ impl MappedMemory for PpMemSpace { self.gen_space.base() as *mut AtomicU64 } - unsafe fn set_bitmap_sink(&self, ptr: *mut u64) { - let st = unsafe { &mut *self.state.get() }; - // Carry the current value across: the boot-time remap runs before the - // CPU thread starts, so banks are typically already mapped by now. - let current = unsafe { *st.sink }; - st.sink = ptr; - unsafe { *ptr = current }; - } - unsafe fn set_bitmap_sink2(&self, ptr: *mut u64) { let st = unsafe { &mut *self.state.get() }; let current = unsafe { *st.sink }; @@ -1354,33 +1591,6 @@ mod tests { assert_eq!(space.mapped_bitmap() & 1, 0, "partial region must read unmapped"); } - /// The bitmap must land in whatever `u64` the CPU handed over, so the hot - /// path reads it as a plain inline field. - #[test] - fn bitmap_publishes_into_the_cpu_field() { - let (space, _banks) = PpMemSpace::with_bank_sizes(&[64]).unwrap(); - - // Stand in for MipsCore.ppmem_bitmap. Boxed so its address is stable. - let cpu_field = Box::new(0u64); - let ptr = &*cpu_field as *const u64 as *mut u64; - - // Map before registering: the boot-time remap runs before the CPU - // thread exists, so the current value must carry across. - space.map_bank(0, 0x0800_0000, 64 * MB as u64, 64 * MB as u64).unwrap(); - let expected = space.mapped_bitmap(); - assert_ne!(expected, 0); - - unsafe { space.set_bitmap_sink(ptr) }; - assert_eq!(*cpu_field, expected, "existing bitmap not carried to the CPU field"); - - // Subsequent changes must publish straight through. - space.map_bank(0, 0x1000_0000, 64 * MB as u64, 64 * MB as u64).unwrap(); - assert_eq!(*cpu_field, space.mapped_bitmap(), "later remap did not publish"); - - space.clear_mappings(); - assert_eq!(*cpu_field, 0, "clear_mappings did not publish"); - } - /// The bit the CPU tests must be the one covering that physical address. #[test] fn cpu_bitmap_test_selects_the_right_region() { @@ -1696,4 +1906,149 @@ mod tests { let gen_bytes = min_bank / GEN_RATIO as usize; assert_eq!(gen_bytes, 16 * 1024, "8MB bank => 16KB of counters"); } + + // ---- cached views ---- + + /// A CPU's view of `space`, publishing into a boxed stand-in for its + /// inline bitmap field. + fn view_of(space: &PpMemSpace) -> (usize, Box) { + let field = Box::new(0u64); + let v = space.add_cached_view(&[&*field as *const u64 as *mut u64]).unwrap(); + (v, field) + } + + fn word(base: *mut u8, at: u64) -> u32 { + unsafe { *(base.add(at as usize) as *const u32) } + } + + fn set_word(base: *mut u8, at: u64, w: u32) { + unsafe { *(base.add(at as usize) as *mut u32) = w }; + } + + /// A view shows every mapping the window has, laid before or after it + /// was made, and publishes the same bitmap to its own sink. + #[test] + fn a_view_follows_the_windows_mappings() { + let (space, banks) = PpMemSpace::with_bank_sizes(&[64, 64]).unwrap(); + space.map_bank(0, 0, 64 * MB as u64, 64 * MB as u64).unwrap(); + let (v, field) = view_of(&space); + space.map_alias(1, 0x0800_0000, 64 * MB as u64).unwrap(); + banks[0].write32(0x40, 0x1111_1111); + banks[1].write32(0x80, 0x2222_2222); + assert_eq!(word(space.view_base(v), 0x40), 0x1111_1111); + assert_eq!(word(space.view_base(v), 0x0800_0080), 0x2222_2222); + assert_eq!(*field, space.mapped_bitmap()); + set_word(space.view_base(v), 0x44, 0x3333_3333); + assert_eq!(banks[0].read32(0x44).data, 0x3333_3333, "an unheld page is memory itself"); + space.clear_mappings(); + assert_eq!(*field, 0); + } + + /// A held page is the view's own: what is written there stays there, the + /// window and memory keep their bytes, and releasing it drops the page. + #[test] + fn a_held_page_is_private_to_its_view_until_released() { + let (space, banks) = PpMemSpace::with_bank_sizes(&[64]).unwrap(); + space.map_bank(0, 0, 64 * MB as u64, 64 * MB as u64).unwrap(); + let (v, _field) = view_of(&space); + banks[0].write32(0x2040, 0x1234_5678); + assert_eq!(space.hold_line_page(v, 0x2040), Some(true)); + assert_eq!(space.hold_line_page(v, 0x2080), Some(false), "a second line of the page"); + let base = space.view_base(v); + assert_eq!(word(base, 0x2040), 0x1234_5678, "it starts as memory"); + set_word(base, 0x2040, 0xaaaa_aaaa); + banks[0].write32(0x2044, 0x5555_5555); + assert_eq!(banks[0].read32(0x2040).data, 0x1234_5678); + assert_eq!(word(space.window_base(), 0x2040), 0x1234_5678); + assert_eq!(word(base, 0x2044), 0, "the page does not see later memory writes"); + assert!(!space.release_line_page(v, 0x2040), "one line left"); + assert_eq!(space.held_line_pages(v), vec![0x2000]); + assert!(space.release_line_page(v, 0x2080)); + assert!(space.held_line_pages(v).is_empty()); + assert_eq!(word(base, 0x2040), 0x1234_5678, "memory again"); + assert_eq!(word(base, 0x2044), 0x5555_5555); + } + + /// Two CPUs holding one page each see their own bytes. + #[test] + fn two_views_hold_one_page_apart() { + let (space, _banks) = PpMemSpace::with_bank_sizes(&[64]).unwrap(); + space.map_bank(0, 0, 64 * MB as u64, 64 * MB as u64).unwrap(); + let (a, _fa) = view_of(&space); + let (b, _fb) = view_of(&space); + space.hold_line_page(a, 0x10_0000); + space.hold_line_page(b, 0x10_0000); + set_word(space.view_base(a), 0x10_0040, 0xaaaa_aaaa); + set_word(space.view_base(b), 0x10_0040, 0xbbbb_bbbb); + assert_eq!(word(space.view_base(a), 0x10_0040), 0xaaaa_aaaa); + assert_eq!(word(space.view_base(b), 0x10_0040), 0xbbbb_bbbb); + assert_eq!(word(space.window_base(), 0x10_0040), 0); + } + + /// A remap lays memory out again under every view and must not take a + /// held page with it. + #[test] + fn a_remap_keeps_held_pages() { + let (space, banks) = PpMemSpace::with_bank_sizes(&[64]).unwrap(); + space.map_bank(0, 0, 64 * MB as u64, 64 * MB as u64).unwrap(); + let (v, field) = view_of(&space); + space.hold_line_page(v, 0x3000); + set_word(space.view_base(v), 0x3010, 0x7777_7777); + space.clear_mappings(); + assert_eq!(*field, 1, "the region the page stands in stays the view's"); + space.map_bank(0, 0, 64 * MB as u64, 32 * MB as u64).unwrap(); + assert_eq!(word(space.view_base(v), 0x3010), 0x7777_7777); + assert_eq!(banks[0].read32(0x3010).data, 0); + space.release_line_page(v, 0x3000); + banks[0].write32(0x3010, 0x9999_9999); + assert_eq!(word(space.view_base(v), 0x3010), 0x9999_9999); + assert_eq!(word(space.view_base(v), 0x200_3010), 0x9999_9999, "laid again with the bank's period"); + } + + /// A page with no memory behind it cannot be held. One in a region the + /// window maps only in part makes the region the view's, reading zeros + /// where nothing is mapped, until the page goes. + #[test] + fn a_page_needs_memory_and_takes_its_region() { + let (space, banks) = PpMemSpace::with_bank_sizes(&[32]).unwrap(); + let (v, field) = view_of(&space); + assert_eq!(space.hold_line_page(v, 0x10_0000), None); + assert_eq!(space.hold_line_page(v, space.window_size()), None, "outside the window"); + space.map_bank(0, 0, 32 * MB as u64, 32 * MB as u64).unwrap(); + assert_eq!((space.mapped_bitmap(), *field), (0, 0), "a 32 MB bank leaves its 64 MB region to the bus"); + assert_eq!(space.hold_line_page(v, 0x40 * MB as u64), None, "nothing behind it"); + banks[0].write32(0x10_0040, 0x600d); + assert_eq!(space.hold_line_page(v, 0x10_0040), Some(true)); + assert_eq!(*field, 1); + assert_eq!(space.mapped_bitmap(), 0, "the shared window is unchanged"); + assert_eq!(word(space.view_base(v), 0x10_0040), 0x600d, "the page starts as memory"); + assert_eq!(word(space.view_base(v), 0x3ff_fffc), 0, "the rest of the region is safe to touch"); + space.release_line_page(v, 0x10_0040); + assert_eq!(*field, 0); + } + + /// A held page's counter carries one `GEN_HELD` however many views hold + /// it, and the mark follows the page to the counter that backs it after + /// a remap, leaving none on the old one. + #[cfg(feature = "jitv2")] + #[test] + fn the_held_mark_follows_the_page() { + use crate::cpu::jitv2::jitv2::GEN_HELD; + let (space, banks) = PpMemSpace::with_bank_sizes(&[64, 64]).unwrap(); + space.map_bank(0, 0, 64 * MB as u64, 64 * MB as u64).unwrap(); + let ((a, _fa), (b, _fb)) = (view_of(&space), view_of(&space)); + let gen = |g: *const AtomicU64| unsafe { (*g).load(Ordering::Relaxed) }; + let at = || unsafe { space.gen_window_base().add(0x2000 >> 12) as *const AtomicU64 }; + space.hold_line_page(a, 0x2000); + space.hold_line_page(b, 0x2040); + assert_eq!(gen(at()) / GEN_HELD, 1, "one mark for two CPUs"); + space.clear_mappings(); + space.map_bank(1, 0, 64 * MB as u64, 64 * MB as u64).unwrap(); + assert_eq!(gen(at()) / GEN_HELD, 1, "on the counter that backs the page now"); + assert_eq!(gen(banks[0].gen_ptr(0x2000)) / GEN_HELD, 0, "and off the old one"); + space.release_line_page(a, 0x2000); + assert_eq!(gen(at()) / GEN_HELD, 1); + space.release_line_page(b, 0x2040); + assert_eq!(gen(at()) / GEN_HELD, 0); + } }