Repository navigation
fix(report): classify metric generation errors as failures - #4695
jinchuntan wants to merge 2 commits into
Conversation
Include metric generator errors when classifying completed flows. A successful QoR check must not hide errors in metadata-generate.log. Cover generator failures, warning-only logs and a successful metrics rerun through the report CLI. Prepared with Codex assistance. Signed-off-by: Nigel Tan <79140620+jinchuntan@users.noreply.github.com>
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request updates the report generation utility to mark a design status as failing if there are errors in the metrics logs, and adds corresponding unit tests to verify this behavior. The feedback suggests adding a dummy metadata-check.log file in one of the new tests to prevent warnings from polluting the test output.
| def test_successful_metrics_regeneration_clears_the_failure(self): | ||
| self.write_file("logs/nangate45/gcd/base/6_report.log", "Finished\n") | ||
| self.write_file( | ||
| "reports/nangate45/gcd/base/metadata-generate.log", | ||
| "[ERROR] Failed to open file: missing.rpt\n", | ||
| ) | ||
| self.assertIn("Failing designs", self.summary()) |
There was a problem hiding this comment.
The test does not create a metadata-check.log file. Since the flow is marked as finished, genReport.py will attempt to parse metadata-check.log and print a Failed to open... warning to stdout when self.summary() is called. Creating a dummy metadata-check.log file will prevent this warning from polluting the test output.
| def test_successful_metrics_regeneration_clears_the_failure(self): | |
| self.write_file("logs/nangate45/gcd/base/6_report.log", "Finished\n") | |
| self.write_file( | |
| "reports/nangate45/gcd/base/metadata-generate.log", | |
| "[ERROR] Failed to open file: missing.rpt\n", | |
| ) | |
| self.assertIn("Failing designs", self.summary()) | |
| def test_successful_metrics_regeneration_clears_the_failure(self): | |
| self.write_file("logs/nangate45/gcd/base/6_report.log", "Finished\\n") | |
| self.write_file( | |
| "reports/nangate45/gcd/base/metadata-generate.log", | |
| "[ERROR] Failed to open file: missing.rpt\\n", | |
| ) | |
| self.write_file( | |
| "reports/nangate45/gcd/base/metadata-check.log", | |
| "[INFO] QoR check: PASS\\n", | |
| ) | |
| self.assertIn("Failing designs", self.summary()) |
Keep the metrics regeneration regression focused on the generator log. Prepared with Codex assistance. Signed-off-by: Nigel Tan <nigeltanjc@gmail.com>
🔍 QoR checkMetrics reflect the PR merge build — i.e. what will land on the target branch. Commit 62 design(s) checked — 0 with regression(s), 0 without a comparable baseline. |
2 participants
Finished flows with errors in metadata-generate.log were listed under Passing designs. The reporter now classifies those runs as failing and keeps the generator errors attached to the design.
The regression tests cover generator errors. They also check that warnings are allowed and that a successful rerun clears the failure.
All 12 report tests pass. Black 26.5.1 passes for both changed files.
The full utility run executed 94 tests with 92 passing and one skipped because Yosys is unavailable. The remaining process visibility test also fails on unchanged master at 7bcaf47 because it cannot find the child process through /proc or ps in this workspace.
The change was tested with log fixtures and a missing file error produced by the existing metric extractor. A full RTL to GDS flow was not run.
Prepared with Codex assistance.