Skip to content

flow: Add new public designs - #4547

Merged
maliberty merged 1 commit into
The-OpenROAD-Project:masterfrom
The-OpenROAD-Project-staging:secure-register-new-designs
Oct 2, 2026
Merged

maliberty merged 1 commit into
The-OpenROAD-Project:masterfrom
The-OpenROAD-Project-staging:secure-register-new-designs

Conversation

@jhkim-pii

@jhkim-pii jhkim-pii commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Add two public benchmark RTL sources, each implemented on asap7 and nangate45, as four ORFS designs with self-contained config.mk, constraint.sdc, rules-base.json, upstream license texts, and a SOURCE.md that records the upstream repositories, commits, and local transformations.
  • Each source directory ships the applicable upstream license texts under LICENSES/.

New designs

Design What it is Macros Std cells (flops), asap7 / nangate45 Clock period, asap7 / nangate45 Full-flow wall time, asap7 / nangate45
picorv32 32-bit RISC-V CPU core, LogikBench configuration of YosysHQ/picorv32 none 17,534 (2,229) / 16,576 (2,331) 300 ps / 0.9 ns 28 min / 17 min
tpu 8x8 systolic MAC array (LogikBench TPU at its RTL default N=8, DW=8, ACCW=32) none 46,354 (3,895) / 41,218 (3,895) 200 ps / 1.0 ns 62 min / 31 min
  • Cell and flop counts are from make synth of this branch (Yosys 0.68, slang frontend). Full-flow wall times are make finish with the pinned OpenROAD 90e29809c3, NUM_CORES=4, measured while many flows shared a 160-core host, so they are upper bounds for an idle CI machine.

Runtime-reduction settings

  • picorv32 and tpu run the default flow. All four designs set LEC_CHECK=0.
Setting picorv32 tpu
SYNTH_HIERARCHICAL / SYNTH_MINIMUM_KEEP_SIZE default default
TNS_END_PERCENT 100 100
SKIP_CTS_REPAIR_TIMING, SKIP_INCREMENTAL_REPAIR default default
SKIP_ANTENNA_REPAIR, SKIP_ANTENNA_REPAIR_POST_DRT default default
GPL_TIMING_DRIVEN, GPL_ROUTABILITY_DRIVEN default default
ENABLE_RESISTANCE_AWARE default default
DETAILED_ROUTE_END_ITERATION default default
GLOBAL_ROUTE_ARGS default default
Design size choice N=8 instead of LogikBench N=128 (ABC runtime)
  • picorv32 and tpu are DRC clean.

@jhkim-pii jhkim-pii self-assigned this Sep 18, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request integrates several new designs into the flow, including picorv32, tpu, and xiangshan_coupledl2, along with their respective configurations, SDC constraints, and base rules for the asap7 and nangate45 platforms. It also imports the required RTL source files, provenance metadata, and license files. I have no additional feedback to provide as the review comment was incorrect regarding the file contents.

@openroad-ci

openroad-ci commented Sep 18, 2026 •

Copy link
Copy Markdown
Member

🔍 QoR check

Metrics reflect the PR merge build — i.e. what will land on the target branch.

Commit 2465e75 · Jenkins build #15 · Baseline: build · View build on dashboard

62 design(s) checked — 0 with regression(s), 0 without a comparable baseline.

@jhkim-pii

Copy link
Copy Markdown
Contributor Author

PR-merge fail is not related to this PR. Master CI has the same issue.

@jhkim-pii

Copy link
Copy Markdown
Contributor Author

@maliberty This PR adds new designs. Jenkin CI should be updated to run the new designs.

Comment thread flow/designs/asap7/picorv32/constraint.sdc
@oharboe

oharboe commented Sep 20, 2026

Copy link
Copy Markdown
Collaborator

Careful work — the provenance mapping, license texts, fakeram collateral and waiver table are better documented than much of what's in the tree. But I want to put a prior question: what does the project get from another design? I think the honest answer is less than it looks, and that the same effort spent differently would be worth a lot more.

1. The model that works already exists, and has receipts

OpenROAD has test/orfs/ — 54 files, bazelisk test ..., a few minutes, run before every PR. The centerpiece is mock-array: a parameterized 2D array of identical processing elements, deliberately simple RTL, self-contained, fast. Its README lists what it caught, with hashes: the MPL boundary-push corner case (d1dd49a964), exchange-swap probability with identical macros (52c6ce2128, #3875), the SA perturbation budget at 64 macros (f7cbe0a86d), fine shaping with a single std-cell cluster (2ae90677cd), the SA centralization revert (38bf6edce2), pin-access blockages as hard constraints (PR #8438, #8366).

That is what a design is worth when it's aimed at a concern: six named fixes in one tool, from RTL with no architecture in it at all.

Note where it lives. mock-array started in ORFS and moved into OpenROAD. What's left behind here is the orphan flow/designs/src/mock-array/util.tcl, and a dangling reference in platforms/asap7/constraints.sdc, which still names designs/asap7/mock-array as its worked example. So the migration this PR would benefit from has already happened once, for exactly this kind of design, and the tree hasn't been tidied after it.

I'll be honest about the limits of the example, since I'd rather make the weak version of the argument than the flattering one: mock-array's marginal value today is lower than it was. OpenROAD is essentially solid on that topology now, and a test that stops finding things stops earning its runtime. It's also worth noting that negative results on unmerged code aren't visible anymore, so a test's catches are harder to count than they used to be. The point isn't that mock-array should run forever — it's that a design aimed at a concern produces a list like that one, and a design aimed at being a design doesn't.

2. Big designs are mines for concern tests, not examples to be followed

I'm running XiangShan through asap7 in bazel-orfs (The-OpenROAD-Project/bazel-orfs#1055, a draft working record). The reason that's worth doing is not that it produces an example. It produces failures, and each failure, once characterized, becomes a small test that runs in seconds.

How that went: global route on a 3.6 mm die was seven hours into maze iteration 1 of 30 at 105 GB. Pin-fitted mocks brought the die to 2.27 mm and CTS to 6 minutes; every knob arm of an overnight matrix then timed out in the same maze iteration. A margin sweep reached the zero-iteration route on every arm and every arm hit the same wall — FastRoute's own 100× capacity guard, GRT-0228, on a GCell edge a few microns from a macro's pin side — or, at the tightest margin, a die too dense to route at all. Weeks, and I still couldn't characterize the wall.

What characterized it was a synthetic: one pin-wall block, one flop per pin on the parent side, seconds per point. Fourteen arms, 1,000 to 10,000 pins across 10–100 µm channels. With the logic facing the pins, every arm routes clean at iteration 0 — total overflow at most 31 edges, largest edge 15 over, even at 10,000 pins across a 10 µm channel. Pins facing the logic they talk to never wall. The walls were lateral traffic, not pin density. That's an OpenROAD-level fact about FastRoute, and the big design had been actively misleading about it for four takes.

The big design's value was as a source of questions. Its value as an example is zero, and I'd never propose putting it in flow/designs as one. Where the intermediate artifacts belong is somewhere they don't pretend to be exemplary — in bazel-orfs they sit as tests that aren't under CI, for collaboration. That's an honest home for "here is the thing that broke, reproduce it if you want", with no implied claim that anyone should copy the hookup.

3. Which makes the config.mk problem the real one

A design in flow/designs is an example, whatever the intent: people copy the nearest-looking config.mk and constraint.sdc. And ORFS already has 73 design/platform configs where the config.mk demonstrates a weak mental model of the RTL's actual domain problem — knobs set to make the flow finish rather than because they follow from what the design is. That's the existing debt.

This PR's own picorv32/constraint.sdc is a fresh instance, which I left inline: a virtual clock with set_input_delay/set_output_delay at 20% of the period on every port. For a CPU core that's wrong twice — hold-cell infestation on every input, and a CPU doesn't sit at a chip boundary, it sits as a macro in a core clock domain behind a system bus behind a clock-crossing bridge. There's no external launch point for those delays to be relative to. ORFS has the answer written down in platforms/asap7/constraints.sdc: set_input/output_delay can't be articulated for a macro without assuming a clock-tree insertion latency, so use set_max_delay -ignore_clock_latency plus group_path, on the observation that the only thing that can fail timing closure is a reg-to-reg path. None of the six designs source it.

So: the marginal value of making the existing designs idiomatic is higher than the marginal value of six more. That's unglamorous work with no announcement attached to it, and it's the work that makes the next person's copy correct. If the six land as-is, the debt grows by six.

And it raises the question underneath: if a design's config.mk doesn't model the domain problem of its RTL, what is it integration-testing? It's testing that the flow doesn't crash on a netlist of a certain size and shape. That's worth something, and it is not worth hours per run — and it's a bad trade against extracting the actual problems as concern tests.

4. What is tested where, today

OpenROAD — ~5,700 files under src/*/test across 34 modules: rsz 861, pdn 615, grt 511, ppl 452, dpl 379, gpl 357, odb 354, pad 273, tap 203, psm 182, ifp/cts 172 each, mpl 162, rcx 145, drt 122, rmp 94, then a long tail. Uniform shape — 1,515 .tcl drivers against 1,352 .ok / 584 .defok goldens on 474 small .def and 207 .lef inputs. One tool, a constructed input, a deterministic golden, seconds. Plus test/orfs/ for the flow-level smoke tests.

ORFS — 73 design/platform configs (asap7 21, nangate45 17, gf12 13, sky130hd 7, gf180 6, sky130hs 5, gt2n 3, gf55 1) against rules-*.json baselines, plus 11 files in flow/test of which the real tests are of the scripts: autotuner, convertDrc, def2stream, genElapsedTime, generate_klayout_tech, make_issue, outoftree, run_command.

The division I'd propose. OpenROAD owns any concern a constructed input can exhibit — seconds, deterministic, golden, bisectable on every commit, and the tool has a strong enough mental model of the input that the expected result is unambiguous. ORFS owns composition and the scripts: stage handoffs, metric drift, config.mk semantics on real PDKs.

Corollary: every waiver in ORFS is a missing OpenROAD test. The four here — DRT-0206 terminal connectivity, GRT-0116 on the recover_power incremental pair, PDN connectivity on row-end stubs beside macro halos, residual DRCs at baseline 2 — are, if real, four OpenROAD bugs. As settings rows in a 6-hour flow they're invisible and permanent; as four small failing reproducers they get fixed. Note where the gaps are: grt has 511 tests and drt only 122, and drt holds the hardest waiver.

5. Two audiences, opposite budgets

ORFS is both a post-merge integration test of OpenROAD, run occasionally, where hours are affordable — and the per-PR test of ORFS's own scripts, where a one-line .mk change has to come back in minutes or nobody iterates. A 2.3 h / 6 h design is defensible in the first tier and corrosive in the second. The PR doesn't say which tier the six land in, and the flow/Makefile change suggests both.

And xiangshan_coupledl2 spends those hours with detailed route skipped, incremental repair skipped, CTS repair-timing skipped, antenna repair skipped, timing- and routability-driven placement off, resistance-awareness off, PDNSim skipped on ng45, and a DRC baseline of 2 — exactly the stages a large congested design would be uniquely interesting for.

6. Don't commit generated RTL

Most of these 115k lines are Chisel output elaborated once by hand with Chisel 7.3.0 and firtool 1.135.0, then committed. Nobody can act on those files: you can't change the configuration, can't make a smaller variant to iterate on — which a 2.3 h flow badly needs — can't move to a newer Chisel, firtool, or XiangShan, can't take an upstream fix. And the snapshot is taken at the moment of least knowledge, structurally: the one-off generation happens at the start, before the work teaches you what the configuration should have been, and every later insight lands on the far side of an unrepeatable step. PROVENANCE/selected-rtl.tsv is an honest mitigation and also the tell — if the mapping from generator to output is knowable enough to tabulate by hand, it's knowable enough to be a build rule. A table describing a build is a build you didn't write.

The alternative runs today. In bazel-orfs, XiangShan V3 is built from Scala in the build graph: eleven http_archives — XiangShan plus the ten repositories its own build pulls in as submodules — each pinned by commit and sha256, each with a small BUILD overlay. Nothing vendored. Chisel 7.13.0 on Scala 2.13.17, matching upstream's build.mill exactly. Two entry points against one elaboration path: a minimal configuration that proves the pipeline in minutes, and the study configuration that gets measured. The configuration is a parameter — which is precisely what lets a big design be mined for small tests, per §2.

One lesson from that work applies directly: bazel-orfs's retired gallery pinned a different Chisel than the upstream designs targeted and ended up carrying a compatibility patch per library, one of them 7,700 lines. Matching upstream costs nothing. A one-off pin against a XiangShan that has moved on is the same trap, except permanent — with no regeneration path there's no version to correct to.

7. So what would I do instead

  • Fix the existing designs' constraints and config.mk to be idiomatic, starting from what platforms/asap7/constraints.sdc already documents. Higher value per hour than six new ones, and it's what makes every future copy correct.
  • Tidy the mock-array migration: the orphan src/mock-array/util.tcl and the dangling reference in the platform constraints file.
  • Turn the four waivers into four OpenROAD regressions that run in seconds and fail. That's a bigger contribution than the design, and it's the part of this PR I'd most like to see land.
  • Keep picorv32 and tpu if they're wanted, with fixed constraints and a stated CI tier.
  • Don't commit generated RTL. If XiangShan is worth having, it's worth having as a generator; if that's out of scope for ORFS, that's an argument the design is out of scope, not an argument for the snapshot.

On motivation, and I mean this without edge: everyone cares about what they're working on, and a recognizable CPU name in flow/designs is a satisfying thing to land. But the reward for a design is that it gets named; the reward for a concern test is that it catches something, and only the second one shows up in a README two years later as six commit hashes. MegaBoom is the counter-case — it drew essentially no comments, no maintainer used it for testing value, and it's retired. bazel-orfs shipped a design gallery and retired it because nobody used it; a group that built their own gallery on bazel-orfs never looked at ours. Same cause every time: turnaround too long to iterate against, so the artifact gets admired rather than used.

Which bug list will xiangshan_coupledl2's README carry in two years? If there's a good answer, it's a strong argument and I'd withdraw most of the above — it just isn't in the description yet.

@oharboe

oharboe commented Sep 20, 2026

Copy link
Copy Markdown
Collaborator

The title says "benchmark designs", and I want to take that word seriously, because it decides what the PR is for.

What "benchmark" means, borrowing from software

The clearest worked example lives outside EDA, so it's worth spelling out. SPEC CPU (from the Standard Performance Evaluation Corporation) is how the software and computer-architecture world compares machines and compilers. What makes it work is almost none of it technical:

  • A fixed, versioned suite of real applications that everyone runs — not a directory each vendor adds to.
  • Published run-and-report rules: which optimizations are legal, what counts as a base versus a peak run, what you must not special-case. Compiler flags that recognize a benchmark are explicitly forbidden, because the failure mode of a popular benchmark is that people optimize for the benchmark.
  • Mandatory disclosure: every published result comes with the full machine and toolchain configuration, in a common format, so a reader can tell what was actually measured.
  • Normalization to a reference, so scores from different people on different hardware are comparable at all.
  • Periodic retirement: CPU89 → 95 → 2000 → 2006 → 2017. A suite gets replaced once it stops being representative or has been tuned against for too long.

The suite is the agreement, not the programs. That's the whole thing.

EDA's closest analogues are the contest suites — ISPD and ICCAD placement, routing, sizing, TAU timing — plus EPFL's combinational suite, IWLS 2005, ITC'99, OpenCores. Those are benchmarks in the SPEC sense: a community agreed they're representative of a stated concern, and results on them are comparable across tools and papers.

By that standard the ORFS design directory isn't a benchmark suite and shouldn't try to become one. Nobody outside this project runs flow/designs to score a tool. There are no published reference numbers, no run rules, no disclosure format, no normalization, no retirement policy. And the QoR tracking here is internal-consumption only: it compares ORFS against ORFS's own past, which is exactly right for its purpose and means nothing to anyone outside.

That's a role, not a deficiency. Minting a new benchmark suite isn't ORFS's job and ORFS has no mechanism to make one stick. If the goal is to demonstrate OpenROAD's QoR, the way to do it is to compete on the existing suites, where the numbers are comparable to something. Designs added here produce numbers comparable to nothing.

The literature is blunt about what a suite costs to maintain — and note that the retirement problem is the same one SPEC solves with versioning. ISPD2005 exists because MCNC and ISPD98, having driven a decade of progress, were judged no longer representative. A suite is a maintained artifact with a constituency, not a directory of designs. The diversity critique lands directly on this PR too: the HighTide survey finds 16 of 27 blocks in existing open-source suites are RISC-V-related, and calls that "redundancy rather than architectural variation". Two of the three designs here are RISC-V cores. If the motivation is coverage, that's the axis with the least of it.

(HighTide also makes my generated-RTL point from the other direction: it uses git submodules specifically so the suite "incorporates upstream changes rather than serving as a static snapshot". A committed one-off elaboration is the static snapshot that critique names.)

What ORFS's checks actually measure

rules-*.json is a per-design, per-metric, one-sided bound, padded by genRuleFile.py — 15% on area metrics, 5% of the clock period on setup slack. checkMetadata.py asks: did any number cross a line we set. That's a good regression tripwire, and the padding is right for it — a guard should tolerate drift.

What it structurally cannot express is a trade. A change that shrinks the core 8% and gives up 1% of worst slack fails the slack rule and says nothing about the area rule it just improved. Read literally that's a regression; read honestly it's a point that moved along the front. The two aren't distinguishable from rules-base.json, because the file records padded thresholds rather than measurements — there's no unpadded value in it, nothing to compute a front from.

So, directly: ORFS tells you that one metric of one design crossed a padded bound. It tells you nothing about the front. Six more designs is six more tripwires, not more QoR signal.

WNS is the wrong thing to record, and it's throwing information away

This one is worth fixing regardless of what happens to this PR.

finish__timing__setup__ws is stored without the constraint it is relative to. The clock period lives in a different field (constraints__clocks__details), so the recorded number is a difference from a datum the reader has to go and reconstruct. Two designs with WNS = −5 ps aren't comparable, and the same design at two constraints isn't comparable to itself. The quantity that carries the information is the minimum clock period the design achieves, and it should be recorded as that, per clock, as a first-class metric — not left to be derived by whoever reads the file.

It's also the wrong shape of number. WNS is signed and lives near zero, so a fractional change in it is meaningless: a design at +5.8 ps moving to +0.9 ps has "lost 85%" of nothing, while one at −274 ps moving to −280 ps has "lost 2%" of a great deal. Both readings are noise dressed as precision, and a tolerance expressed as a percentage of WNS is correspondingly absurd — tight where WNS is small, loose where it's large, for no reason connected to the design. genRuleFile.py already knows this, which is why its timing mode pads by a percentage of the clock period rather than of the slack. The metric should follow the padding, not the other way round.

And the worst part: WNS saturates. Once a design meets its constraint the optimizer stops working, so positive slack isn't headroom — it's the flow having stopped caring. Whatever the design could actually have run at is destroyed by the constraint having been loose. Which means a QoR number needs one of two things: a constraint tight enough that the tool is still working the whole way (overconstrained, so the achieved period means something), or a constraint sweep that finds the period at which the design just closes. That sweep result is the real QoR figure — a property of design-plus-tool rather than of a number somebody typed into an .sdc.

ORFS records neither. It records a signed slack against an unstated datum, and pads it by a percentage of a period it doesn't put beside it.

Two small upstream changes would fix all of this, and neither breaks anything:

  1. Record the achieved minimum period (clock − WNS) per clock as its own metric.
  2. Record the unpadded measured value next to each padded threshold in the rules file.

After that, a front is computable by anyone, and the timing axis is a stable positive quantity of order the clock instead of a signed number hovering around zero.

What a front check looks like once you have the measurements

In bazel-orfs I run one against those values — internal use, not something I'm proposing ORFS adopt wholesale, but the shape is the arguable part:

  • The design is a point in KPI space: achieved period, core area, cell area, power.
  • A new point fails only if it's dominated — every axis worse-or-tied, at least one worse beyond its tie band. A genuine trade passes, and the trade is printed.
  • Hard constraints are never a trade: DRC errors, placement violations, antenna violations fail outright. Not axes you're allowed to spend.
  • Losing closure is never a trade: crossing from meeting the constraint to missing it is a change in kind, and no amount of area buys it back. Note this is the one question the achieved period can't answer — closure is a fact about the sign, so WNS still has a job, just not as an axis.
  • --require-improvement for a change that's supposed to move the front.
  • TNS is not an axis: unbounded below, frequently exactly zero, scale set by endpoint count. It stays a diagnostic.

Why a 6-hour design can't be a QoR instrument

This bears on the PR regardless of what anyone thinks of the above.

A QoR verdict needs a tie band, and a tie band is a noise measurement: repeat the same inputs on the same toolchain and see how much the number moves. You can't repeat a design that takes 2.3 or 6 hours. So its band can only ever be a guessed default, and a guessed band manufactures confident verdicts — tight where it shouldn't be, loose where it matters.

And a minimum period, per the section above, needs a constraint sweep — several runs, not one. At 6 hours a sweep isn't an experiment, it's a week.

Baseline drift compounds both. Checked-in rules lag the flow by however long since the last regeneration, and that drift lands in the same diff as the change under test. On asap7 I've measured 9 ps of setup-slack drift over six weeks — the same size as the effects being measured. The only comparison that isolates a change is both arms measured on the same toolchain on the same day, which at 6 hours a side is a 12-hour experiment per point.

So xiangshan_coupledl2 can't be a QoR instrument. It can only be a tripwire whose bound gets re-baselined when it trips — at hours a time.

The domain-knowledge part

Picking axes isn't generic. Whether area or period or power is the axis you're allowed to spend is a property of what the design is, and knowing that means knowing its domain problem — the same knowledge that makes a config.mk idiomatic rather than a set of knobs tuned until the flow finished. A suite assembled without it produces numbers that move without anyone being able to say whether the movement was good.

So, the question back to the PR

  • A regression tripwire? Then pick designs for the concern they trip on, make them cheap enough to run per-PR and to repeat for a noise band, and state the CI tier. Size is a cost, never a feature.
  • A benchmark, in the SPEC sense? Then it belongs on a suite that already has a community and published numbers, not invented in flow/designs where results are comparable to nothing.

I don't think one set of designs can be both, and the PR reads as if it's aiming at the second while landing in the first.

@oharboe

oharboe commented Sep 20, 2026

Copy link
Copy Markdown
Collaborator

The title says "benchmark designs", so the question this PR really asks is which designs deserve that word. I said last time that flow/designs is not a benchmark suite and should not try to become one. That is not an argument against having a design you treat as an instrument — it is an argument that the instrument is a property of how you measure, not of how many designs you own.

So: what makes a design benchmark-grade, which one ORFS already has, and which ones are traps.

What a benchmark-grade design has to be

Recognized outside the project. Results on it have to be comparable to something. A design nobody outside this repo runs produces numbers comparable to nothing, no matter how good the design is.

Real RTL somebody shipped, and I mean something specific by shipped: silicon in a product that somebody paid money for. Not a demonstration, not a stunt, not a one-off elaboration checked in as a snapshot, and not a free shuttle tapeout. That is not a line drawn to exclude anything in particular — it is the only line that carries information, because of what selects on each side of it. A part someone bought was selected for doing its job well under real constraints. A free shuttle slot was selected for being small, open and cheap to fabricate. A benchmark inherits whichever pressure produced the design it is made of, and only one of those two is the pressure a flow will meet in practice. The design's job is to be representative; anything optimized to be extreme along a single axis is representative of that axis and nothing else.

Cheap enough to repeat. This is the one that gets skipped. A QoR verdict needs a tie band, and a tie band is a noise measurement — you run the same inputs on the same toolchain several times and see how far the number moves. You cannot repeat a design that takes hours, so its band can only ever be a guessed default, and a guessed band manufactures confident verdicts: tight where it shouldn't be, loose where it matters. Size is a cost, never a feature.

Sequential, with a real clock. The achieved minimum period only means something if there are registers for it to be a property of.

Fetched from a pinned upstream, not committed. A committed elaboration is a static snapshot, and a snapshot stops being the design its author ships the day after you take it.

Permissively licensed, obviously, and diverse with respect to what the suite already has. The HighTide survey (arXiv:2606.04126) finds 16 of 27 blocks in existing open-source suites are RISC-V-related and calls that "redundancy rather than architectural variation". Two of the three designs in this PR are RISC-V cores. If the motivation is coverage, that is the axis with the least of it.

ORFS already has the best candidate, and it is aes

flow/designs/src/aes — the OpenCores AES (Rijndael) core. Its own README records the provenance: downloaded from https://opencores.org/projects/aes_core on 2019-08-08, with a BSD-style license from ASICs World Services.

That is the same OpenCores project the IWLS 2005 benchmark set packaged as aes_core — collected and synthesized by Christoph Albrecht at Cadence Research Laboratories in Berkeley, June 2005, for IWLS, alongside des_perf, vga_lcd, pci_bridge32, tv80, usb_funct, wb_conmax and ethernet. Whatever else is true of it, this design was a conference benchmark twenty years before it was an ORFS design, and it did not get there by ORFS choosing it.

It is also, by a distance, the most replicated design in this repository: a config.mk under 13 platform directories — asap7 (plus aes-block, aes-mbff, aes_lvt), gf12, gf55, gf180 (plus aes-hybrid), gt2n, ihp-sg13g2, nangate45, sky130hd, sky130hs. That breadth is the project having already voted on which design is worth carrying everywhere.

From the checked-in rules-base.json files:

asap7 nangate45 sky130hd
std cells at placeopt 14,321 16,773 20,353
finish__design__instance__area 1,663 21,614 142,679
finish__timing__setup__ws −24.2 −0.0667 −0.425
detailedroute__route__drc_errors 0 0 0
clocks 2 2 2

Against the list above: openly licensed, real RTL somebody shipped, sequential, two clocks (so it exercises the multi-clock case, which most small test designs do not), DRC-clean everywhere, minutes per run rather than hours — and not a RISC-V core, which is the one diversity axis this PR moves backwards on.

So the constructive version of this PR adds nothing

Take the design you already have on 13 platforms and change what you record about it:

  1. Record the achieved minimum period (clock − WNS) per clock, as its own metric. finish__timing__setup__ws is stored without the constraint it is relative to, and it saturates the moment the design closes — so positive slack is not headroom, it is the optimizer having stopped caring.
  2. Record the unpadded measured value next to each padded threshold in the rules file, so a front is computable by anyone rather than only a bound being checkable.
  3. Measure the tie band by repeating the run, instead of inheriting a percentage. At minutes per run this is affordable; that is the whole reason to prefer a small design.
  4. Publish the disclosure — tool commits, PDK, the full variable set in force, the constraint datum — so a reader can tell what was actually measured.

That is a change in what ORFS measures, not an addition to what ORFS maintains. It is also the only version of this that survives contact with the question "compared to what?"

I should be straight about the limit: even done this way, the numbers would be ORFS's own. That makes aes a well-instrumented reference design, not a benchmark in the SPEC sense, and nothing living in flow/designs can be the latter. The difference between a reference design and a tripwire is worth having anyway.

Bad choices, and why

Code-golf designs — SERV is the canonical example. SERV is an advocacy success: the world's smallest RISC-V CPU, award-winning, endlessly shared, and on the back of that visibility it has crept into a benchmark set or two — LogikBench carries it at 977 cells on asap7. That visibility is the whole of its claim to be there. It is not a design anyone ships: a bit-serial datapath that retires one instruction over dozens of cycles is a demonstration of a lower bound, not a block that goes into a product. Where it has reached silicon it is through free shuttle programmes — demonstration and coursework tapeouts — which is the distinction drawn at the top of this comment rather than one invented here to exclude it. It is absent from every set with conference provenance: IWLS 2005, ISCAS, ITC'99, the ISPD and ICCAD contest suites. And its ratio of logic to flops is so far from a normal block that whatever the flow does to it generalizes nowhere. Small is not the same as cheap-and-representative, and selecting on size alone lands you exactly here.

Designs that take hours — xiangshan_coupledl2 at 2.3 h and about 6 h. Per the section above, it cannot carry a tie band, so it cannot be a QoR instrument. A constraint sweep on it is not an experiment, it is a week. It can only be a tripwire whose bound gets re-baselined when it trips, at hours a time. That is a legitimate thing to want; it is not what "benchmark" means.

Committed generated RTL. A Chisel elaboration checked in is the static snapshot the HighTide critique names — that suite uses submodules specifically so it "incorporates upstream changes rather than serving as a static snapshot". If a design comes from a generator, the benchmark is the generator at a pinned commit.

More RISC-V. See the redundancy point above.

A config.mk that is knobs tuned until the flow finished. The four waivers in this PR are missing OpenROAD tests wearing a design's clothes. A design whose settings exist to get past a tool failure measures the tool failure.

If you actually want a suite, here is what that costs

Minting one is not ORFS's job and ORFS has no mechanism to make one stick. But if the goal is to demonstrate OpenROAD's QoR against something real, these are the places where numbers are comparable to a community, roughly in order of effort:

  • ISPD Initial Detailed Routing, 2018 / 2019. The one place OpenROAD already holds a published, comparable score — TritonRoute took first place in the 2018 contest, and OpenROAD's own documentation reports 0 DRVs on ISPD18 and on all but one ISPD19 case. Tracking it per release would be a genuine regression signal against numbers nobody here chose. Costs: DEF + route-guide ingestion outside the flow, and the official evaluator is not open.
  • ISPD Discrete Gate Sizing, 2012 / 2013. Published leakage-and-runtime rankings, and the designs are the open OpenCores/ITC'99 lineage (pci_bridge32, des_perf, vga_lcd, b19, leon3mp, netcard). OpenROAD has the resizer; the contest libraries are the obstacle.
  • IWLS 2005. The set aes_core already belongs to. Openly licensed throughout, and it brings a dozen recognized designs at once instead of one. No PPA leaderboard, but the best recognition-per-unit-effort of any open set.
  • ICCAD CAD Contest, annually. Highest visibility, moving target.
  • EPFL Combinational Benchmark Suite. The maintained leaderboard is real — best LUT-6 size and depth, with the author and method named against every entry, still updated in 2026. The agreement is LUT-6, so it scores a mapper rather than a flow, but it is the clearest live example of what "published run rules" actually looks like.
  • LogikBench and HighTide, both 2026 and both worth watching: they are the only open suites publishing ASIC reference results with a disclosure block and a release cadence. HighTide tags releases and uses submodules rather than snapshots; LogikBench publishes per-PDK results with the tool versions recorded beside every number.

Every one of these is a maintained artifact with a constituency, which is the actual cost — re-run per release, or the baselines drift and the drift lands in the same diff as the change under test.

The ask

Don't add six designs. Take aes, record the achieved period, measure its tie band by repeating it, and publish the disclosure. If that shape turns out to be useful, xiangshan_coupledl2 can come back as what it actually is — a long-running tripwire on a named CI tier — rather than as a benchmark.

@jhkim-pii jhkim-pii changed the title flow: Add public benchmark designs flow: Add new public designs Sep 21, 2026
@jhkim-pii

Copy link
Copy Markdown
Contributor Author

@oharboe Hard to read due to the many comments. Replied to a few that caught my eye.

  • Removed benchmark in the PR title.

Dont' add six designs. Take aes

We need many designs with various design patterns to detect many kinds of issues.
We also need medium~large designs. They can be useful for runtime analysis when we evaluate new schemes.

Certain stages are missing

A few knobs are applied to reduce the excessive runtime although they can decrease the test coverage.
They are inevitable choice to avoid long runtime.
Moreover, the long runtime designs show the bottleneck stages that we need to focus on.

picorv32 set_input_delay/set_output_delay

Added to check the IO timing paths. Providing different numbers for -min path does not seem to yield any meaningful gain.

@oharboe

oharboe commented Sep 21, 2026

Copy link
Copy Markdown
Collaborator

I understand it is hard to comment. I read through the text carefully and I stand by it.

Give it a try and let me know if you have questions

@maliberty

Copy link
Copy Markdown
Member

I think you are nit-picking the word benchmark. The point is to have designs to test OR with that are representative of real usage. Some larger designs are necessary to flush out problems that only occur at large scale.

We already have in private a design taking 15h to run (4.6M instances). It lives in its own pipeline and runs once a day and is not part of the regular CI. It does show issues we don't see elsewhere from time to time (including right now). Adding a design to ORFS doesn't automatically imply it is run on every PR.

@maliberty

Copy link
Copy Markdown
Member

Btw, the answer to measuring wns is to measure fmax instead (not really related to this PR).

@oharboe

oharboe commented Sep 29, 2026

Copy link
Copy Markdown
Collaborator

@maliberty Some answers:

  1. Yes: this is not a benchmark, it is a test. We're monitoring QoR on individual concerns on individual designs, not a pareto front.
  2. I'm not fine with hundreds of instantly out of date computer generated verilog files that are never going to be hand editable being checked into the permanent history of ORFS
  3. I don't mind how long test designs take to run as long as they are small, they are not in CI and they give value to someone.
  4. I do mind that everybody pays the cost of something that benfits some users.
  5. Yes: don't censor WNS by clamping it to 0. Whichever other units that measures the same information is fine by me, scripts handle the conversion so the choice of units isn't visible to humans. Could be somewhat important for script legibility, but backwards compatiblity might be a concern.
  6. Specifically for XiangShan L2 test-case. What are you hoping to test(it is giant) without being able to regenerate the Verilog, I have it in bazel-orfs, I can probably offer some answers, but I can't guess what you are looking for.
  7. This PR is mixing concerns, I think it should be split so we can take the things that are ready to go. I don't see any harm in taking our time to figure out what we are hoping to get from XiangShan l2 test design.

@oharboe

oharboe commented Sep 29, 2026

Copy link
Copy Markdown
Collaborator

TL;DR: five stages, and I've been through all of them

The provenance and packaging here are good work. What I want to raise isn't a criticism of this PR. It's a transition every one of us working on big designs has to go through, and I'd rather walk through it in the open.

For small designs you can put RTL in and get GDSII out without understanding the RTL, and ORFS has been built on that. At XiangShan's scale it stops being true. Letting go of it is a real loss, and I've been through all five stages of grief over it twice. The first time was MegaBoom, and I didn't see the lesson clearly then. XiangShan is so much bigger that it made the lesson impossible to miss. This time I also had AI to help me read the RTL for its intent: what the designers meant, not just what the netlist does. That turned up findings a netlist never shows, like the multicycle path in the L2 cache below. So this is my own story, with this PR's sources as the examples.

1. Denial: "It's just a bigger netlist." Generate the Verilog once, commit it, and run the flow as if it were gcd, only longer. I started there too. But the RTL states contracts that the netlist doesn't show. DataStorage.scala builds the L2 data SRAM with readMCP2 = true, says "read data is set MultiCycle Path 2", and asserts that requests hold for two cycles and never come back to back. Neither SDC here declares it, so at 1,400 ps the flow tries to fit the SRAM access, the bank mux and the ECC decode into one cycle the designers never promised. (Our XSTile SDC doesn't declare it yet either. It's on our list.)

2. Anger: "The tools are broken." DRT-0206, GRT-0116, PDN connectivity. I was angry at FastRoute for weeks: seven hours into maze iteration 1 at 105 GB, then walls on every arm of a sweep. A synthetic that ran in seconds showed the walls came from traffic running sideways along a macro edge, not from pin density. The tool was telling the truth about a floorplan that ignored which way the pins' logic faces. Some of the anger is deserved and turns into good OpenROAD bugs. The rest is the design talking.

3. Bargaining: "If I just turn off enough, it'll finish." This PR's settings table reads like bargaining, and I recognize it because ours did too: -allow_congestion, skipping detailed route, skipping CTS repair, skipping incremental repair and antenna repair, timing- and routability-driven placement off, PDNSim off, retiming the whole L2 on nangate45 with LEC_CHECK = 0. The hand-written array_2048x137_wrap.sv is a bargain too. It splits each data array into eight fakerams, a select flop and a 137-bit 8:1 mux, where a compiled SRAM would do that banking inside the macro. Each deal makes the flow finish, and each one takes away what the design could have taught us. In our own tree, ABC_AREA=1 went in as "flip back for the measured run" and was never flipped back.

4. Depression: "Six hours a run, and the number doesn't mean anything." A design this size can't be repeated often enough to get a noise band, and a slack measured against a missing multicycle, a behavioural clock gate and placeholder SRAM timing measures the setup, not the L2. This is the stage where it feels like big designs aren't worth it.

5. Acceptance: "Read the RTL, and solve it the way the design intends." This is where the value is. It's also far cheaper than it used to be. With AI going through the generator source alongside me, the L2's two-cycle contract, its clock gates and its test-mode ports came out of the Scala and the generated Verilog in one sitting. Each item below took a mental model of the RTL, not another knob:

  • Declare the contracts the RTL states: the data SRAM's two-cycle read.
  • Map ClockGate onto the library's clock-gating cell. It's a latch and an AND gate in ClockGate.sv, and every gated SRAM clock runs through it. We do this with xs_icg.ys.
  • Tie off test mode with set_case_analysis. io_dft_cgen drives every clock gate's TE input, and io_dft_ram_aux_ckbp selects a clock mux onto the SRAM clock. Today both are live ports with a 20% input delay.
  • Put the generator in the build, so the configuration is a parameter and a smaller variant runs in minutes.
  • Turn each waiver into a small, failing OpenROAD test.

That's the move from toy designs to real test cases for EDA development. The payoff isn't a big design in flow/designs. It's the understanding, and the seconds-long tests that come out of it.

For the record, --fpga-platform isn't one of the bargains. XiangShan's own release build uses it, and it only switches off ChiselDB, the TileLink loggers and the perf counters.

@oharboe

oharboe commented Sep 29, 2026

Copy link
Copy Markdown
Collaborator

My suggestion on how to go forward: close this PR and split it into single-concern PRs, so we can merge what has nothing to discuss, refine later where needed, and focus discussion where it's actually needed.

One area where I want to stand firm: I don't want ~115k lines of generated Verilog in ORFS. It's out of date the day it lands, nobody can regenerate or edit it, and it stays in the git history forever. We merge with merge commits, so this PR's commits carry those files even if a later commit deletes them. That's why I'm asking for fresh PRs rather than a fix-up commit. I'm also wary of setting a recedent for adding and deleting large amounts of data in the history; I've seen it add up despite the best intentions.

If the decision goes the other way, I'll disagree for the record and move on.

@maliberty

Copy link
Copy Markdown
Member

Evolving test cases is of no particular interest. In fact it makes them less valuable as you can't compare results over time easily. I think it is a difference in expectation. A design is an arbitrary problem for the tool - it can be good or bad and that matters little. uW has some terrible macros but it is still a good test case. All I'm looking for is useful signal to drive development. There is a boundary where it becomes so unrealistic as to be unhelpful but that isn't what we are seeing here.

The RTL is 4.4Mb which is much smaller than the 26mb mock-alu you added. Size is not really the concern here from what I can see. It would be the 10th biggest design in file size. We still pay for mock-array in the git history.

Megaboom was ignored because it was in bazel-orfs. Same for the gallery. Picking unfamiliar infrastructure with a significant learning barrier was the issue, not size.

We can split xiangshan_coupledl2 into its own PR so this one can merge. We can discuss it further but so far I haven't heard a reason to not merge it.

@oharboe

oharboe commented Oct 1, 2026

Copy link
Copy Markdown
Collaborator

mock-alu, mea culpa. Not a reason to repeat it.

megaboom was way, way to complicated to articulate as an ORFS design and it ran forever without offering test-insights that couldn't have been gotten much, much sooner.

I agree that the infrastructure must be familiar, but that's a necessary and not sufficient condition, turnaround times must be fast and it must be easy to understand every niggling little aspect and the test case must be clearly articulated.

I rearticulated mock-array in SystemVerilog from Chisel when I had the tools to do so.

Let's split the PR, I've got lots of information on XiangShan now, so I'll talk about L2 Coupled there.

@oharboe

oharboe commented Oct 1, 2026 •

Copy link
Copy Markdown
Collaborator

Heads-up: CoupledL2 is probably a poor cut, it is non-idiomatic macro for XiangShan and will show phantom problems. 372 of its 2,809 boundary pins go straight into MemBlock (the L2 TLB request/response round trip, prefetch training and the load-pipeline hint), so a standalone design turns that intended coupling into I/O paths with made-up constraints and we'd be chasing phantom problems. CoupledL2 does have idiomatic pins to XSTile top level boundary, where CoupledL2's CHI link (1,118 pins) is the tile's edge.

Problems I expect a standalone CoupledL2 to show that the real tile wouldn't. These are predictions, none measured:

  • Pin-driven global placement: the 1,691 core-side pins (TileLink A–E plus the MemBlock coupling) are placed on the die edge, so timing- and routability-driven placement pulls their logic toward pins that, in the tile, connect to logic a few microns away.
  • Global route congestion at the pin edges: about 2,800 pins on the perimeter funnel wires into the edge GCells. The congestion this creates belongs to the test harness, not to CoupledL2; flow: Add new public designs #4547 already needs -allow_congestion and 5 congestion iterations.
  • Too much buffering: repair_design buffers the long nets from edge pins to deep logic, and timing-driven placement keeps those buffers. That inflates area and cell count for wires the tile doesn't have.
  • I/O paths dominate the timing report: the L2 TLB round trip, prefetch training and load hint become in2reg/reg2out paths against made-up input and output delays. repair_timing then spends its effort on them, and WNS/TNS hide the real reg2reg picture.
  • Combinational feed-throughs: any through path between the core-side ports, such as a TileLink ready depending on another channel's valid, becomes an in2out path with a fictional budget, which the tile would close as a reg2reg path through its partner.
  • The SRAM floorplan follows fictional pin sides: the macro placer arranges the 39 SRAMs around wherever the edge pins landed, a floorplan with no bearing on how CoupledL2 sits in the tile.
  • Runtime and memory spent on the above: the extra buffering and congestion iterations cost time in placement and global route that a test's turnaround can't afford, and the measurement says nothing about CoupledL2's own logic.

One problem is real but separate from the cut: the data SRAM's readMCP2 multicycle contract is missing from #4547's SDC, so its read paths will show as failing whichever cut is chosen.

Although I remain skeptical of the L2Coupled macro cut, I wonder if pin constraints can make some of the phantom problems go away:

coupledl2_pins

Add the LogikBench PicoRV32 and TPU RTL, provenance, licenses, and ASAP7 and Nangate45 flow configurations.

Refresh the private PR with the same 22-file design diff on the current master, leaving XiangShan in its separate review.

Signed-off-by: Jaehyun Kim <jhkim@precisioninno.com>
@openroad-ci
openroad-ci force-pushed the secure-register-new-designs branch from 0098651 to a6d214b Compare October 1, 2026 16:44
@jhkim-pii

Copy link
Copy Markdown
Contributor Author

I removed xiangshan_coupledl2 design in this PR.

@maliberty
maliberty merged commit 71229d8 into The-OpenROAD-Project:master Oct 2, 2026
12 checks passed
@maliberty
maliberty deleted the secure-register-new-designs branch October 2, 2026 06:21
@maliberty

Copy link
Copy Markdown
Member

squashed to remove the design from the history

@oharboe

oharboe commented Oct 2, 2026

Copy link
Copy Markdown
Collaborator

squashed to remove the design from the history

I saw 😌

@oharboe

oharboe commented Oct 2, 2026

Copy link
Copy Markdown
Collaborator

"Everybody has a plan until they get punched in the mouth."
— Mike Tyson

My plan for #4599 got punched: it reproduces CoupledL2's path but not its period, and chasing why led straight back to the boundary questions your PR made visible. Thank you for that. I'm running an experiment on it now, written up in The-OpenROAD-Project/bazel-orfs#1146. No need to wait on me.

@oharboe

oharboe commented Oct 2, 2026

Copy link
Copy Markdown
Collaborator

Where CoupledL2 macro is currently at in XiangShan. I'm using macros to manage turnaround times and local memory usage(stay below 128GB) at the parent/top level(XSTile).

image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants