feat: render the test scenarios you wrote, and only those

README.md

@@ -17,6 +17,13 @@ in dependency order, and writes:
 * the target module's output to `dist/`
 * internal working files (render plan, dependencies, scenarios, requirement lists) to `.pyro/`
 
+The tests come from the test plain files of the spec: one test per scenario, and
+no scenario of pyro's own. By convention a test plain file lives at
+`tests/<module>/<func-id>.plain` next to the spec, and it is bound to its
+functionality by its `module:` and `func_id:` frontmatter. A functionality with
+no test plain file, or one whose file holds no scenario, renders without
+conformance tests and pyro warns about it.
+
 ## Installation
 
 ### Claude Code plugin

skills/render-spec/SKILL.md

@@ -43,23 +43,30 @@ Input arguments are:
 
 Inputs artifacts are:
 * specs of <TargetModule> and the modules it requires or imports, including referenced files inside the specs.
+* the test plain files of those modules, holding the test scenarios of every :plainFunctionality:.
 
 Generated artifacts are:
 * :RenderPlan: is a table listing all the modules for rendering and current render state. :RenderPlan: lives in `./.pyro/render-plan.md`.
 * :plainImplementationCode: lives under `./plain_module/code`. Every module is rendered into this same folder, each building on top of the previously rendered ones.
-* :ConformanceTests: are living under `./plain_module/tests`.
-* Internal working files (dependencies, per-module scenarios and requirements) live under `./.pyro`.
+* :ConformanceTests: and :AcceptanceTests: are living under `./plain_module/tests`.
+* Internal working files (dependencies, per-module requirements) live under `./.pyro`.
 
 ### Available tools and paths
 
 `<python>` is the interpreter resolved in Step 0 - always run them with it, and never through a shell of your own, so the same command line works on macOS, Linux and Windows alike.
 
 `scripts/plain_sections.py` extracts one section out of any number of specs and prints it to stdout:
   `<python> "<skill_folder>/scripts/plain_sections.py" [--include-filename] [--output <path>] <section> <spec.plain> [<spec.plain> ...]`
-  * `<section>` is one of `defs`, `impl-reqs`, `test-reqs`, `func-specs`, `acc-tests`, `all`.
+  * `<section>` is one of `defs`, `impl-reqs`, `test-reqs`, `func-specs`, `conf-tests`, `acc-tests`, `all`.
   * `--output <path>` writes the result to `<path>`.
   * Always gather sections with this script - never transcribe or summarize them by hand.
 
+`scripts/test_scenarios.py` prints the test scenarios of one module to stdout, grouped by func-id in render order:
+  `<python> "<skill_folder>/scripts/test_scenarios.py" <module.plain>`
+  * It binds every test plain file in the `tests/` folder next to the spec to its :plainFunctionality: by the file's `module:` and `func_id:` frontmatter.
+  * Its output is the complete list of :ConformanceTests: and :AcceptanceTests: for the module.
+  * Always gather scenarios with this script - never transcribe or summarize them by hand.
+
 # Step 2: Prepare the :RenderPlan:
 
 Find and read the frontmatter of <TargetModule> and all specs that <TargetModule> requires.
@@ -99,15 +106,16 @@ Before rendering the first module, create the folders `plain_module/code`, `plai
 Load the :RenderPlan: and for every module follow precisely the steps:
 
 ### Step 4.1: Specs, reqs, test scenarios
-Report with a message "Step 4.1: Loading specs, writing reqs and test scenarios for: <module> + <required-or-imported-modules>".
+Report with a message "Step 4.1: Loading specs, collecting reqs and test scenarios for: <module> + <required-or-imported-modules>".
 
 Run the command with helper script, where <specs> are the module's spec followed by the specs of its imported or required modules:
 `<python> "<skill_folder>/scripts/plain_sections.py" all <specs>`
 The output of this command are all the necessary specs for succesfully rendering this module.
 
-Write exhaustive conformance test scenarios for every :plainFunctionality: of the <module>'s spec into `.pyro/<module>/scenarios.md`.
-* Scenarios should exhaustively test every :plainFunctionality: and should include :AcceptanceTests:.
-* Get the <module>'s :AcceptanceTests: with `<python> "<skill_folder>/scripts/plain_sections.py" acc-tests <module spec>` and cover every one of them.
+Run command to collect all the test scenarios of the <module>:
+`<python> "<skill_folder>/scripts/test_scenarios.py" <module.plain>`
+* If the script exits 1, abort the rendering and report its message.
+* Repeat every warning of the script to the user.
 
 Write the lists of requirements using the helper script, where <specs> are module's spec plus the specs of its imported modules:
 * impl. reqs: `<python> "<skill_folder>/scripts/plain_sections.py" impl-reqs <specs> --output .pyro/<module>/impl-reqs.md`
@@ -123,16 +131,16 @@ Implement all :plainFunctionality: of <module> specs while respecting all the re
 
 Implement :UnitTests:.
 
-Implement :ConformanceTests: covering all test scenarios:
-* Read all test scenarios from `.pyro/<module>/scenarios.md`.
-* Read test requirements in `.pyro/<module>/test-reqs.md`.
-* Implement the conformance tests covering all test scenarios and respecting test requirements into the `plain_module/tests` folder.
+Implement :ConformanceTests: and :AcceptanceTests: from the test scenarios of Step 4.1, respecting the test requirements in `.pyro/<module>/test-reqs.md`:
+* Write exactly one test per test scenario from Step 4.1 into the `plain_module/tests` folder, with the scenario-id in the test name.
+* Write the :AcceptanceTests: of a :plainFunctionality: after its :ConformanceTests: pass.
+* Write no test for a :plainFunctionality: marked `> no test plain file` or `> no test scenarios`.
 
 ### Step 4.3: Tests verification
 Report with a message "#Step 4.3: Tests verification of <module>".
 
 If any tests are failing, go back to the implementation step (4.2), debug it and fix implementation code:
-* Run and verify all :ConformanceTests: are passing.
+* Run and verify all :ConformanceTests: and :AcceptanceTests: are passing.
 * Run and verify all :UnitTests: are passing.
 
 ### Step 4.4: Reqs verification
@@ -143,7 +151,7 @@ Read the list `.pyro/<module>/impl-reqs.md` and for every item:
 * Add checkbox with checked/unchecked status to the item.
 
 Read the list `.pyro/<module>/test-reqs.md` and for every item:
-* Review if the conformance tests respect it.
+* Review if the :ConformanceTests: and :AcceptanceTests: respect it.
 * Add checkbox with checked/unchecked status to the item.
 
 IMPORTANT: if any requirement list item is not passing, go back to the implementation (step 4.2), debug and fix it in the code.
@@ -157,7 +165,7 @@ When all modules are rendered do:
 - remove installed dependency artifacts (e.g. `node_modules/`, `.venv/`, `__pycache__/`, vendored packages) in `plain_module/code`, but keep the dependency manifests and lock files so dependencies can be reinstalled later.
 - copy all of the files in the `plain_module/code` folder to the `./dist` folder:
   `<python> "<skill_folder>/scripts/copy_folder.py" plain_module/code dist`
-- prepare a short report on the :plainImplementationCode: and :ConformanceTests:
+- prepare a short report on the :plainImplementationCode: and :ConformanceTests:, naming every :plainFunctionality: that was rendered without :ConformanceTests:
 - present commands to run tests (unit and/or conformance tests)
 - present the command to run the rendered <TargetModule>
 

skills/render-spec/references/plain-lang.md

@@ -4,12 +4,18 @@
 
 ## .plain File Structure
 
-A `.plain` file has a YAML frontmatter section followed by standardized sections (marked with `***section name***` headers):
+There are two kinds of `.plain` file. A **spec plain file** describes a module. A **test plain file** holds the test scenarios of one functionality of that module.
+
+A spec plain file has a YAML frontmatter section followed by standardized sections (marked with `***section name***` headers):
 - `***definitions***` — declares concepts used throughout the specification
 - `***implementation reqs***` — non-functional requirements about how the software should be built
 - `***test reqs***` — requirements for conformance testing
 - `***functional specs***` — describes what the software should do
 
+A test plain file has its own frontmatter and two sections, `***conformance tests***` and `***acceptance tests***`. See *Test Plain Files* below.
+
+Every section marker is at the start of its line. An indented `***...***` line is not a section.
+
 ## Concept Notation
 
 Concepts are the building blocks of .plain specifications. They are written between colons: `:ConceptName:`. Valid characters include letters, digits, plus, minus, dot, and underscore.
@@ -29,8 +35,8 @@ Concepts must be defined in `***definitions***` before being referenced in other
 | `:Implementation:` | The system implementing `:plainFunctionality:` |
 | `:plainImplementationCode:` | The generated implementation code |
 | `:UnitTests:` | Auto-generated unit tests for individual functionalities (configured in `***implementation reqs***`) |
-| `:ConformanceTests:` | Auto-generated tests that verify implementation conforms to specs |
-| `:AcceptanceTest:` / `:AcceptanceTests:` | Tests that validate specific aspects of the implementation |
+| `:ConformanceTests:` | Tests generated from the `***conformance tests***` scenarios of a functionality |
+| `:AcceptanceTest:` / `:AcceptanceTests:` | Tests generated from the `***acceptance tests***` scenarios of a functionality |
 
 ## Definitions Section (`***definitions***`)
 
@@ -42,19 +48,59 @@ A free-form section for any instructions that steer code generation. Common uses
 
 ## Test Reqs Section (`***test reqs***`)
 
-Specifies requirements for conformance testing — test frameworks, execution methods, and testing constraints. It does not refer to unit tests, which are specified in the `***implementation reqs***` section.
+Specifies requirements for conformance testing — test frameworks, execution methods, and testing constraints. It says HOW `:ConformanceTests:` and `:AcceptanceTests:` are built and run; the scenarios in the test plain files say WHAT they check. It does not refer to unit tests, which are specified in the `***implementation reqs***` section.
 
 ## Functional Specs Section (`***functional specs***`)
 
-Describes what the software should do. Each bullet point is a single piece of functionality that will be implemented.
+Describes what the software should do. Each functionality is a `# <func-id>` heading followed by its body — free text that describes the behavior as prose, `- ` bullets, or any other shape. No body line starts with `# ` (that starts the next functionality) or with `***`.
+
+```plain
+***functional specs***
+
+# add-task
+A :User: can add a :Task:. Only valid :Task: items are added.
+
+# delete-task
+A :User: can delete a :Task:.
+```
+
+A func-id is lowercase letters, digits and hyphens (`add-task`, `export-csv-2`, `ctrl-d-exit`) and is unique within its module. It is an identity, not a label: a functionality keeps its func-id for as long as it exists, even when its text changes. The func-id names the functionality everywhere — in its test plain files, in the render plan, in logs, and in the names of the generated tests.
 
 Each functional spec must be limited in complexity. If a spec is too complex, the renderer responds with "Functional spec too complex!" and it must be broken down into smaller specs. Complexity is measured in lines of code - each spec should imply no more than 200 lines of code.
 
-Functional specs are in **sequential order** and rendered incrementally, one by one. When a spec is rendered, only **previous** functional specs are in the renderer's context — those earlier in the list, plus any from `requires` modules; later specs are invisible, so a spec can never reference or anticipate what comes after it. This ordering matters for incremental rendering and for detecting conflicts between specs.
+Functional specs are in **sequential order** and rendered incrementally, one by one. When a spec is rendered, only **previous** functional specs are in the renderer's context — those earlier in the list, plus any from `requires` modules; later specs are invisible, so a spec can never reference or anticipate what comes after it. The order of the headings is the render order; the func-id itself carries no order. This ordering matters for incremental rendering and for detecting conflicts between specs.
+
+## Test Plain Files
+
+The test scenarios of a functionality live in a test plain file of its own. A spec plain file never holds a `***acceptance tests***` section, nested under a functional spec or at the top level.
+
+```plain
+---
+module: task-manager
+func_id: add-task
+---
+
+***conformance tests***
+
+# add-first-task
+A :User: adds a :Task: named "Buy milk" to an empty :TaskList:. :TaskList: then shows one :Task: named "Buy milk".
+
+***acceptance tests***
+
+# add-without-name
+A :User: adds a :Task: with an empty name. :App: rejects it and :TaskList: stays empty.
+```
+
+- The frontmatter has exactly two keys, both required: `module:` is the spec plain file's name without `.plain`, and `func_id:` is the func-id of the functionality the scenarios belong to.
+- Both sections are required and `***conformance tests***` comes first. Both may be empty — the minimal valid file is the frontmatter and the two markers.
+- `***conformance tests***` holds the scenarios of the **expected use** of the functionality: its main path, with valid parameters, the expected caller, and the data present. They gate the functionality: it is done only when every conformance scenario passes.
+- `***acceptance tests***` holds the scenarios of use that is **not expected**: invalid parameters, wrong or missing authentication, missing data. An acceptance scenario is rendered after the conformance tests of its functionality pass.
+- A scenario is a `# <scenario-id>` heading followed by free text that says what is done and what result is expected. A scenario-id has the same syntax as a func-id and is unique for its functionality.
+- Scenarios describe behavior through the observable interface (CLI, HTTP, files, UI). They never name a function, class or file of the implementation, and the generated test never reaches into the implementation's internals.
 
-## Acceptance Tests (`***acceptance tests***`)
+By convention a test plain file lives at `tests/<module>/<func-id>.plain`, with `tests/` next to the spec plain file. The location is convention only: a test plain file is bound to its functionality by its frontmatter, not by its path. More than one file may bind to the same functionality, and their scenarios are merged.
 
-Nested under individual functional specs to specify how to verify correct implementation. They extend conformance tests.
+The scenarios in these files are the complete and only list of tests for the module. The renderer implements them and nothing else — it never invents a scenario, and it never extends one.
 
 ## YAML Frontmatter
 
@@ -68,4 +114,4 @@ The frontmatter is enclosed between `---` markers and supports:
 
 ## Comments
 
-Lines starting with `>` are ignored when rendering:
+Lines starting with `>` are ignored when rendering:

skills/render-spec/scripts/plain_sections.py

@@ -8,9 +8,14 @@
 #   implementation-reqs  impl-reqs   impl    :plainImplementationReqs:
 #   test-reqs                                :plainTestReqs:
 #   functional-specs     func-specs  specs   :plainFunctionality:
+#   conformance-tests    conf-tests          :ConformanceTests:
 #   acceptance-tests     acc-tests           :AcceptanceTests:
 #   all                  all-sections everything   - every section, in file order
 #
+# The first four sections live in a spec plain file and the two test sections in a test plain
+# file; `all` prints whichever of them the given file holds. Every section is top level - a
+# marker that is indented is not a section.
+#
 # The ***section*** marker itself is never printed, except for `all`, which keeps every
 # section's own marker so the sections stay distinguishable. `>` comment lines are always
 # dropped. With --include-filename each body is preceded by a "## <path>" heading.
@@ -52,6 +57,7 @@
   implementation-reqs  impl-reqs  impl    :plainImplementationReqs:
   test-reqs                               :plainTestReqs:
   functional-specs     func-specs specs   :plainFunctionality:
+  conformance-tests    conf-tests         :ConformanceTests:
   acceptance-tests     acc-tests          :AcceptanceTests:
   all                  all-sections everything
 """
@@ -71,6 +77,9 @@
     "func specs": "functional specs",
     "specs": "functional specs",
     "plainfunctionality": "functional specs",
+    "conformance tests": "conformance tests",
+    "conf tests": "conformance tests",
+    "conformancetests": "conformance tests",
     "acceptance tests": "acceptance tests",
     "acc tests": "acceptance tests",
     "acceptancetests": "acceptance tests",
@@ -82,8 +91,6 @@
 COMMENT_RE = re.compile(r"^[ \t]*>")
 BLANK_RE = re.compile(r"^[ \t]*$")
 MARKER_RE = re.compile(r"^[ \t]*\*\*\*(.*)\*\*\*[ \t]*$")
-SPEC_ITEM_RE = re.compile(r"^-[ \t]")
-TRAILING_WS_RE = re.compile(r"[ \t]+$")
 
 
 def byte_sink(stream):
@@ -134,17 +141,6 @@ def marker_name(line):
     return name
 
 
-def indent_of(line):
-    width = 0
-    while width < len(line) and line[width] in " \t":
-        width += 1
-    return width
-
-
-def dedent(line, width):
-    return line[min(indent_of(line), width):]
-
-
 def read_lines(path):
     """Records exactly as awk would see them: split on \\n only, one trailing \\r stripped."""
     handle = open(path, "r", encoding="utf-8", errors="surrogateescape", newline="")
@@ -164,12 +160,6 @@ def extract(path, want, show_path):
     buf = []
     pending = 0  # blank lines held back, so a trailing run never reaches the output
     in_section = False
-    in_fs = False
-    in_block = False
-    block_indent = 0
-    block_content = False
-    parent = ""
-    last_parent = ""
 
     def flush_blanks():
         for _ in range(pending):
@@ -179,49 +169,6 @@ def flush_blanks():
         marker = marker_name(line)
         col0 = line.startswith("***")
 
-        if want == "acceptance tests":
-            if marker is not None and col0:  # a top-level section starts
-                in_fs = marker == "functional specs"
-                in_block = False
-                continue
-            if not in_fs:
-                continue
-
-            if marker == "acceptance tests":  # nested, indented under one spec
-                in_block = True
-                block_indent = indent_of(line)
-                pending = 0
-                block_content = False
-                if parent != "" and parent != last_parent:
-                    if buf:
-                        buf.append("")
-                    buf.append(parent)
-                    last_parent = parent
-                continue
-            if marker is not None:
-                in_block = False
-                continue
-
-            if in_block:
-                if BLANK_RE.match(line):
-                    pending += 1
-                    continue
-                if indent_of(line) >= block_indent:
-                    if COMMENT_RE.match(line):
-                        continue
-                    if block_content:
-                        flush_blanks()
-                    pending = 0  # drop blank lines right after the marker
-                    block_content = True
-                    buf.append(dedent(line, block_indent))
-                    continue
-                in_block = False  # dedented out of the block; reconsider line
-
-            if SPEC_ITEM_RE.match(line):  # the functional spec this block belongs to
-                parent = TRAILING_WS_RE.sub("", line)
-            continue
-
-        # Top-level section mode.
         if marker is not None and col0:
             if want == "all":
                 if buf: