Skip to content

asap7/l2_dir_hit: an L2 directory's hit, broadcast across the slice in one cycle - #4599

Closed
oharboe wants to merge 4 commits into
The-OpenROAD-Project:masterfrom
oharboe:l2-dir-hit
Closed

oharboe wants to merge 4 commits into
The-OpenROAD-Project:masterfrom
oharboe:l2-dir-hit

Conversation

@oharboe

@oharboe oharboe commented Oct 1, 2026

Copy link
Copy Markdown
Collaborator

Draft: the design and its structure are written and under test. The numbers marked pending come from runs on this ORFS master and replace the placeholders below; the PR leaves draft when this design's minimum period at global route replicates XiangShan's.

What

asap7/l2_dir_hit is a new test design with one concern: a register-to-register path that starts at an L2 directory's s3 registers beside its SRAM macros, decides hit and way in one place, and broadcasts the decision across the cache slice in the same cycle. Of that path, how much is repeaters and wire rather than logic?

It is one cycle of the main pipeline of XiangShan's L2, CoupledL2 (XSCache at 300515bc; the sources cite the lines each piece models). In s3 the directory's hit and way steer:

  • MainPipe's meta write and the replacer's update. The arrays are single ported, so either write makes the directory unready, RequestArb withholds its ready from SinkC, and SinkC's task does not fire. That fire is the enable of a 512-bit register reading SinkC's data buffer.
  • The MSHR allocation. A miss takes the lowest idle of 16 MSHRs, with the state the hit decided.
  • The L1 hint queue, for a hit MainPipe answers itself.
  • The data array's request, at the hit's way.

XiangShan's designers name the path in MainPipe.scala: "The combinational logic path of Directory metaAll -> Directory response -> MainPipe judging whether to respond data is too long", and latch the SinkB response to s4 for it. The paths above stay single cycle.

RTL Readable, parameterised SystemVerilog, read by slang (SYNTH_HDL_FRONTEND = slang): the directory, SinkC, MSHRCtl's allocation, CustomL1Hint and MainPipe/RequestArb, one module each, as CoupledL2 has them
SRAMs Behavioural modules in the firtool port convention (RW0_*). AUTO_MEMORIES = 1 turns each into a generated macro at XiangShan's own shapes: two 1024x156 tag halves, a 1024x128 meta array and four 8192x137 data banks
Constraints constraint.sdc sources $PLATFORM_DIR/constraints.sdc. It is a macro, so the boundaries are set_max_delay optimisation targets, with no set_input_delay/set_output_delay, and only reg2reg can fail. The 473 ps period is derived in the file
The two-cycle data array XiangShan builds the data array with readMCP2 = true, and the RTL asserts that its request and write data hold for two cycles and that no request follows a request. constraint.sdc declares that contract with set_multicycle_path; the clock gate's enable is a one-cycle pulse and keeps one cycle. Without it the array's address and read data would be failures the designers never asked the flow to close
Flow ORFS defaults throughout, all three asap7 Vt's, the full flow through final, no waivers. The slice's units stay modules (SYNTH_HIERARCHICAL, SYNTH_KEEP_MODULES), as CoupledL2's own flow keeps them

Results

The minimum clock period is the SDC period minus the worst slack of the reg2reg group, at global route with the route's parasitics.

minimum period at grt worst path logic / repeaters / wire
XiangShan: CoupledL2 as a block of XSTile 1,128 ps directory req_s3 tag to SinkC's buffer read; the MSHR allocation follows at 1,071-1,088 ps 341 ps (24 cells) / 473 ps (35) / 247 ps
this design pending pending pending

pending: the minimum period at final, and the runtime of each stage.

Try this

cd flow
make DESIGN_CONFIG=./designs/asap7/l2_dir_hit/config.mk

Appendix

A. Why

This is the next design in the series that asap7/tage_update (#4595) starts: each class of path whose minimum period XiangShan's flow sets, rather than its RTL, gets its own small, readable, single-concern design whose minimum period at global route replicates XiangShan's. #4595's backlog lists "L2 cache paths. To be characterised." This is the first of them, characterised.

B. How XiangShan's number was measured

XiangShan Kunminghu (aa6b520) in the configuration with a 512 KiB, 8-way L2, built as XSTile with CoupledL2 hardened as one of its blocks, with the pins the tile's floorplan gives it, on this ORFS master and the OpenROAD it pins. CoupledL2's global route runs on its CTS checkpoint with the post-route repair ORFS runs there. The block's flow differs from ORFS's defaults in TNS_END_PERCENT = 1, SKIP_CTS_REPAIR_TIMING = 1, SKIP_LAST_GASP = 1, PLACE_DENSITY = 0.6, ENABLE_DPO = 0, OPENROAD_HIERARCHICAL = 1, routing layers M2-M9, and a global route with -congestion_iterations 0 -allow_congestion. This design runs ORFS's defaults.

The worst path, at the source:

  • Start: Directory.scala:218, req_s3's tag, the operand of the s3 compare (:270).
  • Through: the hit (:270-272), nine BUFx6f in series at fanout 1 to 3, the way select (:295-312), MainPipe's decision (MainPipe.scala:244-262) and its directory writes (:538-606), the directory's single port (Directory.scala:343), RequestArb's ready (RequestArb.scala:151-155).
  • End: SinkC's buffer read register, RegEnable(dataBuf(...), io.task.fire) (SinkC.scala:190), reached through long-wire BUFx16f repeaters, three of them 51-59 ps each.

C. Learning from earlier attempts

#4547 adds CoupledL2's generated Verilog as a test design. As a standalone top level, 372 of its 2,809 pins go to MemBlock in the tile, and the data array's two-cycle contract is not declared, so its I/O paths and its data array dominate its timing for reasons that are the cut's, not CoupledL2's. This design keeps the path that CoupledL2 has inside XSTile and nothing else, registers every boundary, and declares the contract the RTL states.

D. Found on the way

Flat, this design stops in the macro placer with MPL-0045 Cannot find a balanced partitioning for the clusters. One data bank is about 96 % of the area of the cluster around it, so each area-balanced two-way split keeps the bank and nearly every standard cell together and peels off a few hundred cells; after 13 levels no balance constraint can split the remainder. It is deterministic. Keeping the slice's modules, as CoupledL2's flow does, avoids it; the failure itself is OpenROAD's, and is not this design's concern.

🤖 Generated with Claude Code

…n one cycle

A new test design with one concern: a register-to-register path that
starts at the L2 directory's s3 registers beside its SRAM macros,
decides hit and way, and in the same cycle steers the slice: MainPipe's
meta write and through the single-ported directory RequestArb's ready
to SinkC's 512-bit buffer read, the MSHR allocation, the L1 hint queue
and the data array's request.

It is one cycle of XiangShan's CoupledL2 (XSCache at 300515bc), written
as readable, parameterised SystemVerilog read by slang; the sources cite
the Scala lines each piece models. The SRAMs are behavioural modules in
the firtool port convention, which AUTO_MEMORIES turns into macros at
XiangShan's own shapes. constraint.sdc declares the data array's
two-cycle contract, which the RTL states and asserts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Øyvind Harboe <oyvind.harboe@zylin.com>

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces the l2_dir_hit design, which models one cycle of an L2 cache slice's main pipeline based on XiangShan's CoupledL2 cache. It includes the directory, SinkC, RequestArb, MSHR allocation, L1 hint queue, and data array components. A critical correctness bug was identified in l2_dir_hit.sv where the 2-cycle path contract for the data SRAM is broken because control and select signals are gated by the 1-cycle v_s3 signal, causing them to drop prematurely. It is recommended to implement the suggested fix using raw, ungated signals held by v_s3_hold2 to ensure they hold their values for the required two cycles.

Comment thread flow/designs/src/l2_dir_hit/l2_dir_hit.sv
MainPipe decides on the task's bits, which hold for two cycles, and
applies the valid where each consumer takes it (MainPipe.scala:309,
:499-504, :605). The decision here was gated by the one-cycle s3 valid,
so in the hold cycle the data array's enable, write mode, way and write
data changed before the second edge. The refill data came straight from
a port register; it is now read at s2 and registered for s3, as the
refill buffer's response is.

A simulation of DataStorage.scala:124-131's three asserts on the array's
pins, 20,000 cycles of random traffic: 8,696 violations before, none
after, over 5,966 requests and 4,869 writes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Øyvind Harboe <oyvind.harboe@zylin.com>
@oharboe

oharboe commented Oct 1, 2026

Copy link
Copy Markdown
Collaborator Author

@jhkim-pii @maliberty FYI trying to extract idiomatic problems from L2Coupled where it is easy to build a strong mental model of the problem domain with minimum cognitive load.

The read data's s5 destinations are inside the slice (DataStorage.scala:
119-121). As an output register, s5 was drawn to its pin, and the first
global route's worst path was a bank's read data crossing the die to it
on one net: the harness's boundary, not CoupledL2's path. The port now
takes s5 a cycle later.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Øyvind Harboe <oyvind.harboe@zylin.com>
@openroad-ci

openroad-ci commented Oct 1, 2026 •

Copy link
Copy Markdown
Member

🔍 QoR check

Metrics reflect the PR merge build — i.e. what will land on the target branch.

Commit 54fd4bb · Jenkins build #4 · Baseline: build · View build on dashboard

62 design(s) checked — 0 with regression(s), 0 without a comparable baseline.

The first measurement at global route put this design's worst path on
XiangShan's own, directory s3 to SinkC's buffer read, at about half the
period, with 5 logic cells in MainPipe's part where XiangShan has about
14. MainPipe now decides as XiangShan's does: channel A with prefetch,
CMOs and the alias compare (MainPipe.scala:172-265), the snoop classes
of channel B and forwarding (:281-304), the replacer's answer held for
s3 (:237-239), the five meta-write terms (:538-616), and the data
array's ren and wen terms (:484-495). The directory's tag ECC error on
the hit way feeds the snoop decision (Directory.scala:315-319), and
RequestArb takes B between C and A, behind MainPipe's set blocks
(RequestArb.scala:134-160, MainPipe.scala:954-979).

The simulation of DataStorage.scala:124-131's asserts still reports no
violations, over 3,669 requests and 2,101 writes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Øyvind Harboe <oyvind.harboe@zylin.com>
@oharboe

oharboe commented Oct 2, 2026

Copy link
Copy Markdown
Collaborator Author

😞 will need to try something else

@oharboe oharboe closed this Oct 2, 2026
@oharboe

oharboe commented Oct 4, 2026

Copy link
Copy Markdown
Collaborator Author

Closing. The 1,128 ps this test set out to match turned out to be ours, not CoupledL2's: bazel-orfs's macro annealer and its planned pins together stretched SinkC's read-register enable into a 1,360 µm detour. With ORFS's own RTL-MP, CoupledL2 runs at 743 ps on the same path at its global route, and plain ORFS never makes the detour. In XSTile, RTL-MP takes CoupledL2 from 1,321 to 1,008 ps; the tile's worst path is elsewhere in the core.

The 580–711 ps here is about what CoupledL2 gets on a floorplan that doesn't stretch it, so there's no gap left to reproduce. Write-up: The-OpenROAD-Project/bazel-orfs#1146.

@oharboe

oharboe commented Oct 6, 2026

Copy link
Copy Markdown
Collaborator Author

Correction to my comment above: those periods were read with the generated SRAMs untimed from floorplan on (a bazel-orfs setting, since fixed). With them timed, RTL-MP is not faster: CoupledL2 is 1,591 ps with bazel-orfs's annealer and 1,894 ps with RTL-MP, its worst paths into and out of the DataStorage SRAMs. The detour is real geometry from bazel-orfs's floorplan, but it was not CoupledL2's worst problem. Details: The-OpenROAD-Project/bazel-orfs#1177.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants