[migueldalberto]

checkQorMetrics.py (make metadata-check) used to send IsLocal: true with a fixed default job name and compare against that pipeline's latest master build.

The dashboard now splits inline checks in two. This PR moves the script onto that contract.

Default: a checkout. The script sends the commit metadata.json records (--commit / DASHBOARD_COMMIT overrides it), with no JobName, no IsLocal and no branch. The dashboard searches its default baseline pipelines in order. In each, it takes a master build that has the design:

• the build at the commit's merge base with master; a master commit is its own merge base, so a downloaded master artifact compares against its own build;
• otherwise the latest master build before the commit, which also covers an unpushed commit;
• with no commit recorded, the latest master build. The report now warns when this happens.

--job-name / DASHBOARD_JOB_NAME: a build of that pipeline. The script sends the name plus the branch, from --branch / DASHBOARD_BRANCH or git rev-parse --abbrev-ref HEAD. The dashboard applies that pipeline's CI rules: previous build on a trunk branch, merge base otherwise. CI takes this path and passes the branch and commit.

--base-commit: still pins the baseline. It no longer needs a job name, because the dashboard searches its default pipelines for the pinned build.

Other changes

• --api-url / DASHBOARD_API_URL picks the deployment and takes precedence over --beta.
• Non-finite metric values (inf/NaN) are dropped and listed in the report. json.dumps rendered them as bare Infinity/NaN, which the dashboard rejects with HTTP 400 for the whole request.
• The report names the pipeline that provided the baseline.

[gemini-code-assist]

Code Review

This pull request updates the QoR metrics check to compare runs against the master build at their merge base by default, rather than the latest master build. It also introduces support for new environment variables and command-line arguments (--api-url, --commit, and --branch), filters out non-finite metrics to prevent dashboard errors, and adds a helper to resolve the current git branch. The review feedback highlights a potential issue where cleaning the commit SHA using a simple regular expression can corrupt it when ref prefixes or git describe tags are present, and suggests a more robust parsing approach.

[openroad-ci]

🔍 QoR check

Metrics reflect the PR merge build — i.e. what will land on the target branch.

Commit 77a6ad7 · Jenkins build #6 · Baseline: build · View build on dashboard

62 design(s) checked — 0 with regression(s), 0 without a comparable baseline.

[vvbandeira]

Items 1 and 2 should be addressed before merge; 3-7 are small items and nits that you can address or ignore.

1. A commit the dashboard does not know falls back to current master without a warning.
   When the dashboard has no record of the commit, it uses the current time as the cutoff and returns latest-master.
   render_run warns about that only when no commit was sent (checkQorMetrics.py:410).
   So a local commit that is not pushed is compared against today's master.
   If its base is older, upstream changes show up as false regressions or hide real ones, and the result changes as master moves.
   Metrics.md:55-56, the module docstring and the utils.mk comment say the opposite.
   Please warn on latest-master whenever it is returned, suggest --base-commit for an exact baseline, and fix the docs.
   Also update the Metrics.md intro (lines 11-12), which still says "the latest master build by default".
2. No tests.
   Payload modes, --commit validation, branch resolution and non-finite filtering decide which baseline is used.
   A flow/test/test_checkQorMetrics.py that stubs urlopen and checks the request, report and exit code would run in the existing util-test workflow.
3. A bad --commit returns before write_log (line 723), so a previous sweep's inline-check.log stays.
4. Non-finite metrics:
   • they are listed before the --only-prefix filter (line 203), so metadata-check-synth warns about metrics it never sends;
   • the warning is lost on the no_baseline path (line 391);
   • a file with only non-finite values reports "no numeric metrics" without naming them.
5. DASHBOARD_JOB_NAME used to be the way to pick a private pipeline (e.g. OpenROAD-flow-scripts-Private).
   It now selects CI branch rules.
   Please say what private users should do now.
6. --base-commit is not checked for a full SHA, while --commit is.
7. Nits:
   • a detached HEAD is logged as INFO, while an empty branch warns;
   • --branch without --job-name is ignored silently;
   • docstring lines 8 and 164 need re-wrapping.

[migueldalberto]

Thanks for the thorough review. I've addressed every point, one commit each.

1. Unknown commit falls back to current master (216034b). You were right: an unpushed commit has no merge base on GitHub and isn't known to the dashboard, so it was compared against today's master. I went a step further than the warning:

• When no remote branch in the checkout contains the run's commit, the script computes git merge-base origin/master itself and pins the baseline to that build.
• A commit that is on a remote, or not in the checkout at all (a downloaded CI artifact), is still resolved by the dashboard.
• It's on by default. DASHBOARD_MERGE_BASE / --merge-base picks another ref, and none turns it off.
• It's skipped with --base-commit or --job-name. CI checkouts often fetch no remote branches, so every commit there would look unpushed.
• Every latest-master result now warns and points at DASHBOARD_BASE_COMMIT, a new env var for --base-commit so that make metadata-check can reach it.
• If a pinned merge base has no build of the design, the report suggests DASHBOARD_MERGE_BASE=none.
• Metrics.md (the intro and the resolution section), the module docstring and the utils.mk comment now describe the fallback as it really works.

2. Tests (f1f341e). I added flow/test/test_checkQorMetrics.py, which the util-test workflow picks up. It has 48 tests by the end of the branch. They call run_check() with urlopen stubbed, and check the payload, the report and the exit code. They cover:

• the payload modes and the env vars;
• --commit and --base-commit validation;
• branch resolution, including a detached HEAD;
• the merge-base pin, using a temporary repository with a fake origin/master, so the state of the checkout running the tests doesn't matter;
• metric filtering;
• every verdict, plus retries and --strict;
• the sweep mode.

3. Stale inline-check.log (f69f684). In sweep mode, a rejected --commit now writes its error to the log, replacing the previous report. When there's no reports/ directory, no log is created.

4. Non-finite metrics (f573df5).

• --only-prefix now applies first, so metadata-check-synth names only metrics it would have sent.
• The warning now also appears on the no_baseline path.
• A file whose values are all non-finite now gives an error that names them.
• A prefix that matches nothing now says so.

5. DASHBOARD_JOB_NAME for private pipelines (9a4eb60). The dashboard's default baseline pipelines include the private ones, and their builds are visible only to a caller with a key. So a local run of a private platform needs only DASHBOARD_API_KEY. The docs now say that, say that DASHBOARD_JOB_NAME is meant for CI and is no longer how you pick a private pipeline, and note that it turns off the merge-base pin.

6. --base-commit validation (1ef6243). --base-commit and DASHBOARD_BASE_COMMIT are now checked like --commit: lowercased, and rejected with exit 1 unless they're a full SHA. Before this, a short SHA silently came back as no_baseline and exited 0.

7. Nits (51c7b16).

• A detached HEAD with --job-name now warns, the same as an empty branch.
• --branch without --job-name now warns that it's ignored.
• Both docstring lines are re-wrapped (line 8 in 216034b).