feat: render the test scenarios you wrote, and only those by NejcS · Pull Request #46 · plainlang/pyro

NejcS · 2026-10-30T08:46:29Z

Pyro now renders the test scenarios that you wrote, and only those.

What pyro did before

Pyro wrote the test scenarios itself. It read the spec and decided what to test. Two renders of the same spec could test different things.

What pyro does now

You write the test scenarios. Each functionality has its own test file next to the spec. Pyro reads those files and writes one test per scenario. It adds no scenario of its own.

A test file has two parts. The first part (conformance scenarios) holds the expected use of the functionality. The second part (acceptance scenarios) holds use that is not expected, such as a wrong parameter or missing data. Pyro renders the second part only after the first part passes.

When a test file is missing

A functionality can have no test file yet. Pyro renders it, writes a warning, and gives it no conformance tests. The render goes on.

A test file that points to a functionality that does not exist stops the render. Pyro names the file and asks you to fix it. Those scenarios were written for something, so losing them without notice is worse than stopping.

Old specs stop the render

An old spec stops the render with a message that says so. In the old shape each functionality was one bullet and its tests sat under it. There is no automatic migration.

Before this works

Your spec needs the new shape: an id for each functionality, and a test file for each one. Plain-forge writes this shape today. Users get the change with the next pyro release.

coderabbitai · 2026-10-30T08:46:39Z

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

Source mode: Auto
Repository: Unavailable
Branch: Unavailable
File: No configuration file from this repository applied
Commit: Unavailable
Layers: defaults
Review profile: CHILL
Plan: Advanced
Run ID: 7090e3f0-f9da-45f2-81e3-da07743223e2

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

Trigger review

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

hisenb3rg

I've left few comments - mostly about the skill.md

hisenb3rg · 2026-10-30T11:24:43Z

+`scripts/test_scenarios.py` collects the test scenarios of one module from its test plain files and prints them grouped by func-id, in render order:
+  `<python> "<skill_folder>/scripts/test_scenarios.py" [--output <path>] <module spec.plain>`
+  * It reads the `# <func-id>` headings of the module's `***functional specs***`, scans the `tests/` folder next to the spec, and binds every test plain file to its functionality by the file's `module:` and `func_id:` frontmatter.
+  * Its output is the complete and only list of tests for the module.


I think this line is redundand

hisenb3rg · 2026-10-30T11:28:21Z


+`scripts/test_scenarios.py` collects the test scenarios of one module from its test plain files and prints them grouped by func-id, in render order:
+  `<python> "<skill_folder>/scripts/test_scenarios.py" [--output <path>] <module spec.plain>`
+  * It reads the `# <func-id>` headings of the module's `***functional specs***`, scans the `tests/` folder next to the spec, and binds every test plain file to its functionality by the file's `module:` and `func_id:` frontmatter.


Let's reduce this to simple description of what it does, let's not explain the instructions - they're for this reason coded in the script. Also would avoid *** syntax, to be consistent, and doesn't help the llm...

hisenb3rg · 2026-10-30T11:35:06Z

-* Scenarios should exhaustively test every :plainFunctionality: and should include :AcceptanceTests:.
-* Get the <module>'s :AcceptanceTests: with `<python> "<skill_folder>/scripts/plain_sections.py" acc-tests <module spec>` and cover every one of them.
+Collect the test scenarios of every :plainFunctionality: of the <module> into `.pyro/<module>/scenarios.md`:
+`<python> "<skill_folder>/scripts/test_scenarios.py" --output .pyro/<module>/scenarios.md <module spec>`


Let's not write this to this file, because it is basically just a copy. But rather "run test_scenarios" so that they are printed on top of the context of the llm. We want to force the llm to put this inside the context, and this is the main reason for this script (just telling it read it is not reliable).

hisenb3rg · 2026-10-30T11:35:46Z

-* Get the <module>'s :AcceptanceTests: with `<python> "<skill_folder>/scripts/plain_sections.py" acc-tests <module spec>` and cover every one of them.
+Collect the test scenarios of every :plainFunctionality: of the <module> into `.pyro/<module>/scenarios.md`:
+`<python> "<skill_folder>/scripts/test_scenarios.py" --output .pyro/<module>/scenarios.md <module spec>`
+* The scenarios come from the module's test plain files and from nowhere else. Never write, extend, split or invent a scenario, not even an obvious missing case.


I don't think that necessary to instruct outside of plain-forge (where we are writing specs).

hisenb3rg · 2026-10-30T11:37:58Z

+* If the script exits 1, abort the rendering and report its message: the spec is in a syntax pyro no longer renders, or a test plain file is bound to no :plainFunctionality:.
+* The script warns about every :plainFunctionality: that has no test plain file, or whose test plain file holds no scenario. Repeat every warning to the user. Those functionalities are rendered without :ConformanceTests:.


This would fall into plain-parser, once pyro start using it... so we can keep it.

hisenb3rg · 2026-10-30T11:41:27Z

 * Read test requirements in `.pyro/<module>/test-reqs.md`.
-* Implement the conformance tests covering all test scenarios and respecting test requirements into the `plain_module/tests` folder.
+* Write exactly one test per scenario into the `plain_module/tests` folder, respecting the test requirements. Every test carries the scenario-id of the scenario it implements in its name, so a failing test leads back to the scenario text.
+* The scenarios under `***conformance tests***` of a :plainFunctionality: are its :ConformanceTests:. The scenarios under `***acceptance tests***` are its :AcceptanceTests:, implemented after the :ConformanceTests: of that same :plainFunctionality: pass.


Lets use :AcceptanceTests: and :ConformanceTests: only, no three dots syntax.

hisenb3rg · 2026-10-30T11:43:43Z

 * Read all test scenarios from `.pyro/<module>/scenarios.md`.
 * Read test requirements in `.pyro/<module>/test-reqs.md`.
-* Implement the conformance tests covering all test scenarios and respecting test requirements into the `plain_module/tests` folder.
+* Write exactly one test per scenario into the `plain_module/tests` folder, respecting the test requirements. Every test carries the scenario-id of the scenario it implements in its name, so a failing test leads back to the scenario text.


We don't want to refer at this point to plain_module/tests but to test scenarios or :AcceptanceTests: or :ConformanceTests: that were loaded on the top of the context by "test_scenarios.py". We don't want llm deciding what file to read or not to read here...

Actually this comment didn't make sense - I mixed it with the scenarios.md (input, not output)...

plain_sections.py reads ***conformance tests*** and ***acceptance tests*** as top-level sections, the way a test plain file holds them. It gains the section `conformance tests` (aliases conf-tests, :ConformanceTests:), and drops the walk over ***acceptance tests*** blocks nested under a functional spec: an indented marker is no longer a section, so a spec in the old syntax yields no acceptance tests. The fixtures move to the new syntax: functional specs are `# <func-id>` headings, and the scenarios live under tests/<module>/<func-id>.plain. old-syntax.plain keeps a spec in the old syntax to test against. BREAKING CHANGE: Specs in the old syntax no longer render. Functional specs are `# <func-id>` headings, and test scenarios move out of nested ***acceptance tests*** blocks into test plain files at tests/<module>/<func-id>.plain, under top-level ***conformance tests*** and ***acceptance tests*** sections. The renderer no longer writes tests of its own: a functionality gets only the scenarios its test plain file lists.

test_scenarios.py reads the `# <func-id>` headings of a module's ***functional specs***, scans the tests/ folder next to the spec, binds each test plain file to a functionality by its `module:` and `func_id:` frontmatter, merges the files bound to the same functionality and prints the scenarios grouped by func-id in render order. That output is the complete and only list of tests for the module, so the renderer stops writing scenarios of its own. A functionality with no test plain file, or one whose file holds no scenario, is a warning: the module still renders, that functionality just gets no conformance tests. A spec in the syntax pyro no longer renders - bullets, or a nested ***acceptance tests*** block - stops the run with a message that says so.

Step 4.1 of the skill told the renderer to write exhaustive conformance scenarios of its own. It now collects them with test_scenarios.py and writes one test per scenario, carrying the scenario-id in the test name. The acceptance scenarios of a functionality are implemented after its conformance tests pass. A functionality that the scenarios file marks as having no test plain file, or no scenario, gets no test and is named in the final report. The language reference describes the two kinds of .plain file, functional specs as `# <func-id>` headings, and the test plain file with its frontmatter and its two sections.

A test plain file whose `func_id:` names no functionality of its module is an error now, not a warning. Its scenarios were written for something - a renamed or deleted functionality, or a typo - and rendering on would throw them away with only a warning line to show for it. A functionality with no test plain file stays a warning: those scenarios were never written, which is a state a project is allowed to be in.

NejcS marked this pull request as ready for review October 30, 2026 09:23

NejcS requested a review from hisenb3rg October 30, 2026 10:57

hisenb3rg reviewed Oct 30, 2026

View reviewed changes

hisenb3rg force-pushed the feat/explicit-test-scenarios branch from 8b6f660 to 72d00da Compare November 2, 2026 12:50

NejcS added 4 commits November 2, 2026 07:55

hisenb3rg force-pushed the feat/explicit-test-scenarios branch from 72d00da to e5f0069 Compare November 2, 2026 12:56

hisenb3rg merged commit 97de281 into dev Nov 2, 2026
7 checks passed

Navigation Menu

Repository navigation

feat: render the test scenarios you wrote, and only those - #46

feat: render the test scenarios you wrote, and only those#46
hisenb3rg merged 4 commits into
devfrom
feat/explicit-test-scenarios

NejcS commented Oct 30, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented Oct 30, 2026 •

edited

Loading

Review skipped

Uh oh!

hisenb3rg left a comment

Uh oh!

hisenb3rg Oct 30, 2026

Uh oh!

hisenb3rg Oct 30, 2026

Uh oh!

hisenb3rg Oct 30, 2026

Uh oh!

hisenb3rg Oct 30, 2026

Uh oh!

hisenb3rg Oct 30, 2026

Uh oh!

hisenb3rg Oct 30, 2026

Uh oh!

hisenb3rg Oct 30, 2026

Uh oh!

hisenb3rg Nov 2, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		* If the script exits 1, abort the rendering and report its message: the spec is in a syntax pyro no longer renders, or a test plain file is bound to no :plainFunctionality:.
		* The script warns about every :plainFunctionality: that has no test plain file, or whose test plain file holds no scenario. Repeat every warning to the user. Those functionalities are rendered without :ConformanceTests:.

Conversation

NejcS commented Oct 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coderabbitai Bot commented Oct 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

hisenb3rg left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

NejcS commented Oct 30, 2026 •

edited

Loading

coderabbitai Bot commented Oct 30, 2026 •

edited

Loading