Repository navigation
Keep full-flow steps when generating metadata without QoR checks - #4683
Conversation
Allow an external QoR checker to own the verdict without skipping simulation, power, or issue packaging after metadata generation. Signed-off-by: Vitor Bandeira <vvbandeira@precisioninno.com>
Use a public platform and synthetic timing values in the report fixture. Keep the same assertions for the failed metric, limit, and baseline link. Signed-off-by: Vitor Bandeira <vvbandeira@precisioninno.com>
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request adds support for the finish metadata-generate target in test_helper.sh, which keeps all full-flow follow-up steps and delegates QoR checking to an external caller. It also introduces a new test suite in test_test_helper.py to verify helper targets and adds a test for dashboard QoR check failures in test_genReport.py. Feedback suggests filtering os.environ in the new test helper to ensure tests are hermetic and not affected by host environment variables.
| env = dict( | ||
| os.environ, | ||
| TARGET=target, | ||
| MAKE_ISSUE="1", | ||
| PRIVATE_DIR=str(flow / "private"), | ||
| ) |
There was a problem hiding this comment.
Inheriting the entire os.environ without filtering can make the test non-hermetic and prone to flakiness if variables like WORK_HOME, FLOW_VARIANT, or other flow-related environment variables are set in the host environment. Filtering these out ensures the test runs in a clean, controlled environment.
| env = dict( | |
| os.environ, | |
| TARGET=target, | |
| MAKE_ISSUE="1", | |
| PRIVATE_DIR=str(flow / "private"), | |
| ) | |
| env = { | |
| k: v for k, v in os.environ.items() | |
| if k not in { | |
| "TARGET", "MAKE_ISSUE", "PRIVATE_DIR", "WORK_HOME", | |
| "FLOW_VARIANT", "DESIGN_CONFIG", "DESIGN_NAME", | |
| "PLATFORM", "SAVE_TO_DB", "RUN_CALIBRE", "CHECK_DRC_DB" | |
| } | |
| } | |
| env.update({ | |
| "TARGET": target, | |
| "MAKE_ISSUE": "1", | |
| "PRIVATE_DIR": str(flow / "private"), | |
| }) |
🔍 QoR checkMetrics reflect the PR merge build — i.e. what will land on the target branch. Commit 62 design(s) checked — 0 with regression(s), 0 without a comparable baseline. |
2 participants
The flow helper returns before simulation, power, and issue packaging when a caller selects
finish metadata-generate.Keep those follow-up steps for this target so an external dashboard check can own the QoR verdict.
The default
finish metadataand other custom targets keep their current behavior.Add regression tests for the target's follow-up sequence, failure status, and dashboard metric details in the final summary.
Local validation: 91 utility tests ran with one existing Yosys skip, Bash syntax passed, and the changed Python tests passed Black 26.5.1.
Independent review found no major or blocker issue.
One minor follow-up remains to isolate inherited environment variables in the new helper test.
Risk: land this helper support before a Jenkins caller switches to
finish metadata-generate.