diff --git a/README.md b/README.md index 6501aad..86eb2dc 100644 --- a/README.md +++ b/README.md @@ -17,6 +17,13 @@ in dependency order, and writes: * the target module's output to `dist/` * internal working files (render plan, dependencies, scenarios, requirement lists) to `.pyro/` +The tests come from the test plain files of the spec: one test per scenario, and +no scenario of pyro's own. By convention a test plain file lives at +`tests//.plain` next to the spec, and it is bound to its +functionality by its `module:` and `func_id:` frontmatter. A functionality with +no test plain file, or one whose file holds no scenario, renders without +conformance tests and pyro warns about it. + ## Installation ### Claude Code plugin diff --git a/skills/render-spec/SKILL.md b/skills/render-spec/SKILL.md index 3bd1a8c..d1a5e22 100644 --- a/skills/render-spec/SKILL.md +++ b/skills/render-spec/SKILL.md @@ -43,12 +43,13 @@ Input arguments are: Inputs artifacts are: * specs of and the modules it requires or imports, including referenced files inside the specs. +* the test plain files of those modules, holding the test scenarios of every :plainFunctionality:. Generated artifacts are: * :RenderPlan: is a table listing all the modules for rendering and current render state. :RenderPlan: lives in `./.pyro/render-plan.md`. * :plainImplementationCode: lives under `./plain_module/code`. Every module is rendered into this same folder, each building on top of the previously rendered ones. -* :ConformanceTests: are living under `./plain_module/tests`. -* Internal working files (dependencies, per-module scenarios and requirements) live under `./.pyro`. +* :ConformanceTests: and :AcceptanceTests: are living under `./plain_module/tests`. +* Internal working files (dependencies, per-module requirements) live under `./.pyro`. ### Available tools and paths @@ -56,10 +57,16 @@ Generated artifacts are: `scripts/plain_sections.py` extracts one section out of any number of specs and prints it to stdout: ` "/scripts/plain_sections.py" [--include-filename] [--output ]
[ ...]` - * `
` is one of `defs`, `impl-reqs`, `test-reqs`, `func-specs`, `acc-tests`, `all`. + * `
` is one of `defs`, `impl-reqs`, `test-reqs`, `func-specs`, `conf-tests`, `acc-tests`, `all`. * `--output ` writes the result to ``. * Always gather sections with this script - never transcribe or summarize them by hand. +`scripts/test_scenarios.py` prints the test scenarios of one module to stdout, grouped by func-id in render order: + ` "/scripts/test_scenarios.py" ` + * It binds every test plain file in the `tests/` folder next to the spec to its :plainFunctionality: by the file's `module:` and `func_id:` frontmatter. + * Its output is the complete list of :ConformanceTests: and :AcceptanceTests: for the module. + * Always gather scenarios with this script - never transcribe or summarize them by hand. + # Step 2: Prepare the :RenderPlan: Find and read the frontmatter of and all specs that requires. @@ -99,15 +106,16 @@ Before rendering the first module, create the folders `plain_module/code`, `plai Load the :RenderPlan: and for every module follow precisely the steps: ### Step 4.1: Specs, reqs, test scenarios -Report with a message "Step 4.1: Loading specs, writing reqs and test scenarios for: + ". +Report with a message "Step 4.1: Loading specs, collecting reqs and test scenarios for: + ". Run the command with helper script, where are the module's spec followed by the specs of its imported or required modules: ` "/scripts/plain_sections.py" all ` The output of this command are all the necessary specs for succesfully rendering this module. -Write exhaustive conformance test scenarios for every :plainFunctionality: of the 's spec into `.pyro//scenarios.md`. -* Scenarios should exhaustively test every :plainFunctionality: and should include :AcceptanceTests:. -* Get the 's :AcceptanceTests: with ` "/scripts/plain_sections.py" acc-tests ` and cover every one of them. +Run command to collect all the test scenarios of the : +` "/scripts/test_scenarios.py" ` +* If the script exits 1, abort the rendering and report its message. +* Repeat every warning of the script to the user. Write the lists of requirements using the helper script, where are module's spec plus the specs of its imported modules: * impl. reqs: ` "/scripts/plain_sections.py" impl-reqs --output .pyro//impl-reqs.md` @@ -123,16 +131,16 @@ Implement all :plainFunctionality: of specs while respecting all the re Implement :UnitTests:. -Implement :ConformanceTests: covering all test scenarios: -* Read all test scenarios from `.pyro//scenarios.md`. -* Read test requirements in `.pyro//test-reqs.md`. -* Implement the conformance tests covering all test scenarios and respecting test requirements into the `plain_module/tests` folder. +Implement :ConformanceTests: and :AcceptanceTests: from the test scenarios of Step 4.1, respecting the test requirements in `.pyro//test-reqs.md`: +* Write exactly one test per test scenario from Step 4.1 into the `plain_module/tests` folder, with the scenario-id in the test name. +* Write the :AcceptanceTests: of a :plainFunctionality: after its :ConformanceTests: pass. +* Write no test for a :plainFunctionality: marked `> no test plain file` or `> no test scenarios`. ### Step 4.3: Tests verification Report with a message "#Step 4.3: Tests verification of ". If any tests are failing, go back to the implementation step (4.2), debug it and fix implementation code: -* Run and verify all :ConformanceTests: are passing. +* Run and verify all :ConformanceTests: and :AcceptanceTests: are passing. * Run and verify all :UnitTests: are passing. ### Step 4.4: Reqs verification @@ -143,7 +151,7 @@ Read the list `.pyro//impl-reqs.md` and for every item: * Add checkbox with checked/unchecked status to the item. Read the list `.pyro//test-reqs.md` and for every item: -* Review if the conformance tests respect it. +* Review if the :ConformanceTests: and :AcceptanceTests: respect it. * Add checkbox with checked/unchecked status to the item. IMPORTANT: if any requirement list item is not passing, go back to the implementation (step 4.2), debug and fix it in the code. @@ -157,7 +165,7 @@ When all modules are rendered do: - remove installed dependency artifacts (e.g. `node_modules/`, `.venv/`, `__pycache__/`, vendored packages) in `plain_module/code`, but keep the dependency manifests and lock files so dependencies can be reinstalled later. - copy all of the files in the `plain_module/code` folder to the `./dist` folder: ` "/scripts/copy_folder.py" plain_module/code dist` -- prepare a short report on the :plainImplementationCode: and :ConformanceTests: +- prepare a short report on the :plainImplementationCode: and :ConformanceTests:, naming every :plainFunctionality: that was rendered without :ConformanceTests: - present commands to run tests (unit and/or conformance tests) - present the command to run the rendered diff --git a/skills/render-spec/references/plain-lang.md b/skills/render-spec/references/plain-lang.md index 6954058..f47dc10 100644 --- a/skills/render-spec/references/plain-lang.md +++ b/skills/render-spec/references/plain-lang.md @@ -4,12 +4,18 @@ ## .plain File Structure -A `.plain` file has a YAML frontmatter section followed by standardized sections (marked with `***section name***` headers): +There are two kinds of `.plain` file. A **spec plain file** describes a module. A **test plain file** holds the test scenarios of one functionality of that module. + +A spec plain file has a YAML frontmatter section followed by standardized sections (marked with `***section name***` headers): - `***definitions***` — declares concepts used throughout the specification - `***implementation reqs***` — non-functional requirements about how the software should be built - `***test reqs***` — requirements for conformance testing - `***functional specs***` — describes what the software should do +A test plain file has its own frontmatter and two sections, `***conformance tests***` and `***acceptance tests***`. See *Test Plain Files* below. + +Every section marker is at the start of its line. An indented `***...***` line is not a section. + ## Concept Notation Concepts are the building blocks of .plain specifications. They are written between colons: `:ConceptName:`. Valid characters include letters, digits, plus, minus, dot, and underscore. @@ -29,8 +35,8 @@ Concepts must be defined in `***definitions***` before being referenced in other | `:Implementation:` | The system implementing `:plainFunctionality:` | | `:plainImplementationCode:` | The generated implementation code | | `:UnitTests:` | Auto-generated unit tests for individual functionalities (configured in `***implementation reqs***`) | -| `:ConformanceTests:` | Auto-generated tests that verify implementation conforms to specs | -| `:AcceptanceTest:` / `:AcceptanceTests:` | Tests that validate specific aspects of the implementation | +| `:ConformanceTests:` | Tests generated from the `***conformance tests***` scenarios of a functionality | +| `:AcceptanceTest:` / `:AcceptanceTests:` | Tests generated from the `***acceptance tests***` scenarios of a functionality | ## Definitions Section (`***definitions***`) @@ -42,19 +48,59 @@ A free-form section for any instructions that steer code generation. Common uses ## Test Reqs Section (`***test reqs***`) -Specifies requirements for conformance testing — test frameworks, execution methods, and testing constraints. It does not refer to unit tests, which are specified in the `***implementation reqs***` section. +Specifies requirements for conformance testing — test frameworks, execution methods, and testing constraints. It says HOW `:ConformanceTests:` and `:AcceptanceTests:` are built and run; the scenarios in the test plain files say WHAT they check. It does not refer to unit tests, which are specified in the `***implementation reqs***` section. ## Functional Specs Section (`***functional specs***`) -Describes what the software should do. Each bullet point is a single piece of functionality that will be implemented. +Describes what the software should do. Each functionality is a `# ` heading followed by its body — free text that describes the behavior as prose, `- ` bullets, or any other shape. No body line starts with `# ` (that starts the next functionality) or with `***`. + +```plain +***functional specs*** + +# add-task +A :User: can add a :Task:. Only valid :Task: items are added. + +# delete-task +A :User: can delete a :Task:. +``` + +A func-id is lowercase letters, digits and hyphens (`add-task`, `export-csv-2`, `ctrl-d-exit`) and is unique within its module. It is an identity, not a label: a functionality keeps its func-id for as long as it exists, even when its text changes. The func-id names the functionality everywhere — in its test plain files, in the render plan, in logs, and in the names of the generated tests. Each functional spec must be limited in complexity. If a spec is too complex, the renderer responds with "Functional spec too complex!" and it must be broken down into smaller specs. Complexity is measured in lines of code - each spec should imply no more than 200 lines of code. -Functional specs are in **sequential order** and rendered incrementally, one by one. When a spec is rendered, only **previous** functional specs are in the renderer's context — those earlier in the list, plus any from `requires` modules; later specs are invisible, so a spec can never reference or anticipate what comes after it. This ordering matters for incremental rendering and for detecting conflicts between specs. +Functional specs are in **sequential order** and rendered incrementally, one by one. When a spec is rendered, only **previous** functional specs are in the renderer's context — those earlier in the list, plus any from `requires` modules; later specs are invisible, so a spec can never reference or anticipate what comes after it. The order of the headings is the render order; the func-id itself carries no order. This ordering matters for incremental rendering and for detecting conflicts between specs. + +## Test Plain Files + +The test scenarios of a functionality live in a test plain file of its own. A spec plain file never holds a `***acceptance tests***` section, nested under a functional spec or at the top level. + +```plain +--- +module: task-manager +func_id: add-task +--- + +***conformance tests*** + +# add-first-task +A :User: adds a :Task: named "Buy milk" to an empty :TaskList:. :TaskList: then shows one :Task: named "Buy milk". + +***acceptance tests*** + +# add-without-name +A :User: adds a :Task: with an empty name. :App: rejects it and :TaskList: stays empty. +``` + +- The frontmatter has exactly two keys, both required: `module:` is the spec plain file's name without `.plain`, and `func_id:` is the func-id of the functionality the scenarios belong to. +- Both sections are required and `***conformance tests***` comes first. Both may be empty — the minimal valid file is the frontmatter and the two markers. +- `***conformance tests***` holds the scenarios of the **expected use** of the functionality: its main path, with valid parameters, the expected caller, and the data present. They gate the functionality: it is done only when every conformance scenario passes. +- `***acceptance tests***` holds the scenarios of use that is **not expected**: invalid parameters, wrong or missing authentication, missing data. An acceptance scenario is rendered after the conformance tests of its functionality pass. +- A scenario is a `# ` heading followed by free text that says what is done and what result is expected. A scenario-id has the same syntax as a func-id and is unique for its functionality. +- Scenarios describe behavior through the observable interface (CLI, HTTP, files, UI). They never name a function, class or file of the implementation, and the generated test never reaches into the implementation's internals. -## Acceptance Tests (`***acceptance tests***`) +By convention a test plain file lives at `tests//.plain`, with `tests/` next to the spec plain file. The location is convention only: a test plain file is bound to its functionality by its frontmatter, not by its path. More than one file may bind to the same functionality, and their scenarios are merged. -Nested under individual functional specs to specify how to verify correct implementation. They extend conformance tests. +The scenarios in these files are the complete and only list of tests for the module. The renderer implements them and nothing else — it never invents a scenario, and it never extends one. ## YAML Frontmatter @@ -68,4 +114,4 @@ The frontmatter is enclosed between `---` markers and supports: ## Comments -Lines starting with `>` are ignored when rendering: \ No newline at end of file +Lines starting with `>` are ignored when rendering: diff --git a/skills/render-spec/scripts/plain_sections.py b/skills/render-spec/scripts/plain_sections.py index 7f2b8c3..43be9e4 100644 --- a/skills/render-spec/scripts/plain_sections.py +++ b/skills/render-spec/scripts/plain_sections.py @@ -8,9 +8,14 @@ # implementation-reqs impl-reqs impl :plainImplementationReqs: # test-reqs :plainTestReqs: # functional-specs func-specs specs :plainFunctionality: +# conformance-tests conf-tests :ConformanceTests: # acceptance-tests acc-tests :AcceptanceTests: # all all-sections everything - every section, in file order # +# The first four sections live in a spec plain file and the two test sections in a test plain +# file; `all` prints whichever of them the given file holds. Every section is top level - a +# marker that is indented is not a section. +# # The ***section*** marker itself is never printed, except for `all`, which keeps every # section's own marker so the sections stay distinguishable. `>` comment lines are always # dropped. With --include-filename each body is preceded by a "## " heading. @@ -52,6 +57,7 @@ implementation-reqs impl-reqs impl :plainImplementationReqs: test-reqs :plainTestReqs: functional-specs func-specs specs :plainFunctionality: + conformance-tests conf-tests :ConformanceTests: acceptance-tests acc-tests :AcceptanceTests: all all-sections everything """ @@ -71,6 +77,9 @@ "func specs": "functional specs", "specs": "functional specs", "plainfunctionality": "functional specs", + "conformance tests": "conformance tests", + "conf tests": "conformance tests", + "conformancetests": "conformance tests", "acceptance tests": "acceptance tests", "acc tests": "acceptance tests", "acceptancetests": "acceptance tests", @@ -82,8 +91,6 @@ COMMENT_RE = re.compile(r"^[ \t]*>") BLANK_RE = re.compile(r"^[ \t]*$") MARKER_RE = re.compile(r"^[ \t]*\*\*\*(.*)\*\*\*[ \t]*$") -SPEC_ITEM_RE = re.compile(r"^-[ \t]") -TRAILING_WS_RE = re.compile(r"[ \t]+$") def byte_sink(stream): @@ -134,17 +141,6 @@ def marker_name(line): return name -def indent_of(line): - width = 0 - while width < len(line) and line[width] in " \t": - width += 1 - return width - - -def dedent(line, width): - return line[min(indent_of(line), width):] - - def read_lines(path): """Records exactly as awk would see them: split on \\n only, one trailing \\r stripped.""" handle = open(path, "r", encoding="utf-8", errors="surrogateescape", newline="") @@ -164,12 +160,6 @@ def extract(path, want, show_path): buf = [] pending = 0 # blank lines held back, so a trailing run never reaches the output in_section = False - in_fs = False - in_block = False - block_indent = 0 - block_content = False - parent = "" - last_parent = "" def flush_blanks(): for _ in range(pending): @@ -179,49 +169,6 @@ def flush_blanks(): marker = marker_name(line) col0 = line.startswith("***") - if want == "acceptance tests": - if marker is not None and col0: # a top-level section starts - in_fs = marker == "functional specs" - in_block = False - continue - if not in_fs: - continue - - if marker == "acceptance tests": # nested, indented under one spec - in_block = True - block_indent = indent_of(line) - pending = 0 - block_content = False - if parent != "" and parent != last_parent: - if buf: - buf.append("") - buf.append(parent) - last_parent = parent - continue - if marker is not None: - in_block = False - continue - - if in_block: - if BLANK_RE.match(line): - pending += 1 - continue - if indent_of(line) >= block_indent: - if COMMENT_RE.match(line): - continue - if block_content: - flush_blanks() - pending = 0 # drop blank lines right after the marker - block_content = True - buf.append(dedent(line, block_indent)) - continue - in_block = False # dedented out of the block; reconsider line - - if SPEC_ITEM_RE.match(line): # the functional spec this block belongs to - parent = TRAILING_WS_RE.sub("", line) - continue - - # Top-level section mode. if marker is not None and col0: if want == "all": if buf: diff --git a/skills/render-spec/scripts/test_scenarios.py b/skills/render-spec/scripts/test_scenarios.py new file mode 100755 index 0000000..2bfb62d --- /dev/null +++ b/skills/render-spec/scripts/test_scenarios.py @@ -0,0 +1,329 @@ +#!/usr/bin/env python3 +# Collect the test scenarios of one module from its test plain files. +# +# Usage: test_scenarios.py [--output ] +# +# The scenarios of a functionality live in a test plain file, never in the spec plain file. A +# test plain file carries the frontmatter keys `module:` and `func_id:` and the two sections +# ***conformance tests*** and ***acceptance tests***, each holding `# ` scenarios. +# +# This script reads the `# ` headings of the module's ***functional specs***, scans +# `/tests` for every .plain file, binds each one to a functionality by its +# frontmatter, merges the scenarios of all files bound to the same functionality, and prints +# them grouped by func-id in render order. The output is the complete and only list of tests +# the renderer implements for the module - it never invents a scenario of its own. +# +# A functionality with no test plain file, or one whose file holds no scenario, is a warning: +# the module still renders, that functionality just gets no conformance tests. A test plain file +# whose `func_id:` names no functionality is an error instead, because its scenarios were written +# and the render would throw them away - a renamed or deleted functionality, or a typo. +# +# --output (or --output=) writes the result to instead of stdout, creating +# missing parent folders, for the same reason plain_sections.py has it: a Windows PowerShell +# redirect would hand later steps UTF-16LE. Warnings and errors stay on stderr. +# +# Portability: the standard library only, Python 3.8+, no shell - the same command line works +# from sh, PowerShell and cmd alike. Output is written as bytes to keep it UTF-8 with LF +# endings on every platform. +# +# Exit: 0 ok (missing or empty test plain files only warn), 1 a spec in the syntax pyro no +# longer renders, a malformed func-id, a test plain file bound to no functionality, an +# unreadable file or an unwritable --output path, 2 usage error. + +import os +import re +import sys + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) + +import plain_sections # noqa: E402 - the sibling script in this same folder +from plain_sections import byte_sink, extract, read_lines, write_text # noqa: E402 + +PROG = os.path.basename(sys.argv[0]) + +USAGE = """usage: %(prog)s [--output ] +""" + +# A func-id or a scenario-id: lowercase letters, digits and hyphens. +ID_RE = re.compile(r"^[a-z0-9-]+$") +HEADING_RE = re.compile(r"^#[ \t]+(.*?)[ \t]*$") +FRONT_KEY_RE = re.compile(r"^([A-Za-z_][A-Za-z0-9_]*)[ \t]*:[ \t]*(.*?)[ \t]*$") + +TESTS_FOLDER = "tests" + + +def error(message): + write_text(byte_sink(sys.stderr), "%s: error: %s\n" % (PROG, message)) + + +def warning(message): + write_text(byte_sink(sys.stderr), "%s: warning: %s\n" % (PROG, message)) + + +def usage(): + write_text(byte_sink(sys.stderr), USAGE % {"prog": PROG}) + sys.exit(2) + + +def show(path, start): + """`path` relative to `start`, with forward slashes, so it reads the same on Windows.""" + try: + return os.path.relpath(path, start).replace(os.sep, "/") + except ValueError: # a different drive on Windows + return path.replace(os.sep, "/") + + +def frontmatter(path): + """The `key: value` pairs of the YAML frontmatter; {} when the file has none.""" + lines = read_lines(path) + if not lines or lines[0].strip() != "---": + return {} + pairs = {} + for line in lines[1:]: + if line.strip() == "---": + return pairs + match = FRONT_KEY_RE.match(line) + if match is not None: + pairs[match.group(1)] = match.group(2).strip("'\"") + return {} # no closing marker: the file has no usable frontmatter + + +def scenarios(body, where): + """[(scenario-id, [body lines])] in file order, from a section body or None.""" + found = [] + if body is None: + return found + for line in body.split("\n"): + match = HEADING_RE.match(line) + if match is not None: + found.append((match.group(1), [])) + elif found: + found[-1][1].append(line) + elif line != "": + warning("%s: text before the first `# ` heading is ignored" % where) + for _, body in found: # the blank lines that separated the scenarios + while body and body[-1] == "": + body.pop() + return found + + +def func_ids(spec): + """The func-ids of the module's ***functional specs***, in render order.""" + body = extract(spec, "functional specs", False) + if body is None: + warning("no functional specs in %s" % spec) + return [] + + lines = body.split("\n") + for line in lines: + if plain_sections.marker_name(line) == "acceptance tests": + error( + "%s holds a nested ***acceptance tests*** block. pyro renders the current " + "syntax only: every functionality is a `# ` heading and its scenarios " + "live in a test plain file under %s/." % (spec, TESTS_FOLDER) + ) + return None + + found = [] + for line in lines: + match = HEADING_RE.match(line) + if match is None: + continue + if ID_RE.match(match.group(1)) is None: + error("%s: `# %s` is not a func-id (lowercase letters, digits and hyphens)" + % (spec, match.group(1))) + return None + if match.group(1) in found: + error("%s: the func-id `%s` is used twice" % (spec, match.group(1))) + return None + found.append(match.group(1)) + + if not found: + error( + "%s has no `# ` heading in ***functional specs***. pyro renders the " + "current syntax only: every functionality is a `# ` heading and its " + "scenarios live in a test plain file under %s/." % (spec, TESTS_FOLDER) + ) + return None + return found + + +def test_plain_files(folder): + """Every .plain file under `folder`, sorted, so the merge order is the same everywhere.""" + found = [] + for root, dirs, names in os.walk(folder): + dirs.sort() + for name in sorted(names): + if name.endswith(".plain"): + found.append(os.path.join(root, name)) + return found + + +def bind(spec, module, ids): + """{func-id: [(path, conformance, acceptance)]}, or None when a file is bound to nothing.""" + folder = os.path.join(os.path.dirname(os.path.abspath(spec)), TESTS_FOLDER) + bound = dict((func_id, []) for func_id in ids) + if not os.path.isdir(folder): + return bound + + stale = False + for path in test_plain_files(folder): + where = show(path, os.path.dirname(os.path.abspath(spec))) + front = frontmatter(path) + if "module" not in front or "func_id" not in front: + warning("%s has no `module:` and `func_id:` frontmatter; skipped" % where) + continue + if front["module"] != module: + continue # a test plain file of another module + if front["func_id"] not in bound: + # Its scenarios were written for something: a renamed or deleted functionality, or + # a typo. Rendering on would throw them away without anyone noticing. + error("%s names `%s`, which is not a functionality of %s. Point `func_id:` at the " + "functionality the scenarios belong to, or delete the file." + % (where, front["func_id"], module)) + stale = True + continue + bound[front["func_id"]].append(( + where, + scenarios(extract(path, "conformance tests", False), where), + scenarios(extract(path, "acceptance tests", False), where), + )) + return None if stale else bound + + +def section_lines(marker, merged): + out = ["", marker] + for scenario_id, body in merged: + out.append("") + out.append("# " + scenario_id) + out.extend(body) + return out + + +def merge(module, func_id, files): + """The output block of one functionality, and whether it got any scenario.""" + out = ["## " + func_id] + if not files: + warning("%s:%s has no test plain file; it renders without conformance tests" + % (module, func_id)) + out.append("") + out.append("> no test plain file") + return out, False + + out.append("") + out.append("> from " + ", ".join(path for path, _, _ in files)) + + merged = {"***conformance tests***": [], "***acceptance tests***": []} + seen = {} + for path, conformance, acceptance in files: + for marker, found in (("***conformance tests***", conformance), + ("***acceptance tests***", acceptance)): + for scenario_id, body in found: + if ID_RE.match(scenario_id) is None: + warning("%s: `# %s` is not a scenario-id (lowercase letters, digits and " + "hyphens)" % (path, scenario_id)) + if scenario_id in seen: + warning("%s:%s has the scenario-id `%s` twice, in %s and %s; the second one " + "is skipped" % (module, func_id, scenario_id, seen[scenario_id], path)) + continue + seen[scenario_id] = path + merged[marker].append((scenario_id, body)) + + if not seen: + warning("%s:%s has no test scenario; it renders without conformance tests" + % (module, func_id)) + out.append("> no test scenarios") + return out, False + + for marker in ("***conformance tests***", "***acceptance tests***"): + if merged[marker]: + out.extend(section_lines(marker, merged[marker])) + return out, True + + +def report(module, ids, bound): + out = ["# Test scenarios of " + module, ""] + out.append("> The complete list of tests to implement for this module. The renderer") + out.append("> implements these scenarios and no others; it never invents one.") + for func_id in ids: + out.append("") + block = merge(module, func_id, bound[func_id])[0] + out.extend(block) + return "\n".join(out) + "\n" + + +def parse_args(argv): + out = None + positional = [] + args = list(argv) + while args: + arg = args.pop(0) + if arg == "--output": + if not args: + error("--output requires a path") + usage() + out = args.pop(0) + elif arg.startswith("--output="): + out = arg[len("--output="):] + if out == "": + error("--output requires a path") + usage() + elif arg in ("-h", "--help"): + usage() + else: + positional.append(arg) + return out, positional + + +def main(argv): + out, positional = parse_args(argv) + if len(positional) != 1: + usage() + + spec = positional[0] + if os.path.isdir(spec) or not os.access(spec, os.R_OK): + error("cannot read %s" % spec) + return 1 + + try: + ids = func_ids(spec) + except (IOError, OSError): + error("cannot read %s" % spec) + return 1 + if ids is None: + return 1 + + module = os.path.basename(spec) + if module.endswith(".plain"): + module = module[:-len(".plain")] + + try: + bound = bind(spec, module, ids) + except (IOError, OSError): + error("cannot read the test plain files of %s" % module) + return 1 + if bound is None: + return 1 + + text = report(module, ids, bound) + + if out is None: + write_text(byte_sink(sys.stdout), text) + return 0 + try: + parent = os.path.dirname(out) + if parent and not os.path.isdir(parent): + os.makedirs(parent) + handle = open(out, "wb") + except (IOError, OSError): + error("cannot write %s" % out) + return 1 + try: + write_text(handle, text) + finally: + handle.close() + return 0 + + +if __name__ == "__main__": + sys.exit(main(sys.argv[1:])) diff --git a/tests/expected/acc-tests.txt b/tests/expected/acc-tests.txt index f2a0ae1..d7c6326 100644 --- a/tests/expected/acc-tests.txt +++ b/tests/expected/acc-tests.txt @@ -1,6 +1,2 @@ -- Create a :Widget: from a name. -- Creating a :Widget: named "a" yields a :Widget: with name "a". -- An empty name is rejected. - -- List all :Widget: in the :Registry:. -- An empty :Registry: lists nothing. +# reject-empty-name +An empty name is rejected and the :Registry: stays unchanged. diff --git a/tests/expected/all.txt b/tests/expected/all.txt index 65de281..5bb5e74 100644 --- a/tests/expected/all.txt +++ b/tests/expected/all.txt @@ -17,17 +17,11 @@ ***functional specs*** -- Create a :Widget: from a name. +# create-widget +A :Widget: is created from a name. - ***acceptance tests*** +# list-widgets +All :Widget: items in the :Registry: are listed. - - Creating a :Widget: named "a" yields a :Widget: with name "a". - - An empty name is rejected. - -- List all :Widget: in the :Registry:. - - ***acceptance tests*** - - - An empty :Registry: lists nothing. - -- Delete a :Widget: by name. +# delete-widget +A :Widget: is deleted by name. diff --git a/tests/expected/conf-tests.txt b/tests/expected/conf-tests.txt new file mode 100644 index 0000000..9925e74 --- /dev/null +++ b/tests/expected/conf-tests.txt @@ -0,0 +1,5 @@ +# create-named-widget +Creating a :Widget: named "a" yields a :Widget: with name "a". + +# create-second-widget +A second :Widget: named "b" joins "a" in the :Registry:. diff --git a/tests/expected/func-specs.txt b/tests/expected/func-specs.txt index 8238b32..5678b70 100644 --- a/tests/expected/func-specs.txt +++ b/tests/expected/func-specs.txt @@ -1,14 +1,8 @@ -- Create a :Widget: from a name. +# create-widget +A :Widget: is created from a name. - ***acceptance tests*** +# list-widgets +All :Widget: items in the :Registry: are listed. - - Creating a :Widget: named "a" yields a :Widget: with name "a". - - An empty name is rejected. - -- List all :Widget: in the :Registry:. - - ***acceptance tests*** - - - An empty :Registry: lists nothing. - -- Delete a :Widget: by name. +# delete-widget +A :Widget: is deleted by name. diff --git a/tests/expected/scenarios.txt b/tests/expected/scenarios.txt new file mode 100644 index 0000000..24994ce --- /dev/null +++ b/tests/expected/scenarios.txt @@ -0,0 +1,30 @@ +# Test scenarios of sample + +> The complete list of tests to implement for this module. The renderer +> implements these scenarios and no others; it never invents one. + +## create-widget + +> from tests/sample/create-widget.plain + +***conformance tests*** + +# create-named-widget +Creating a :Widget: named "a" yields a :Widget: with name "a". + +# create-second-widget +A second :Widget: named "b" joins "a" in the :Registry:. + +***acceptance tests*** + +# reject-empty-name +An empty name is rejected and the :Registry: stays unchanged. + +## list-widgets + +> from tests/sample/list-widgets.plain +> no test scenarios + +## delete-widget + +> no test plain file diff --git a/tests/fixtures/old-syntax.plain b/tests/fixtures/old-syntax.plain new file mode 100644 index 0000000..0d8ef7f --- /dev/null +++ b/tests/fixtures/old-syntax.plain @@ -0,0 +1,14 @@ +--- +description: Fixture spec in the syntax that pyro no longer renders. +--- + +***functional specs*** + +- Create a :Widget: from a name. + + ***acceptance tests*** + + - Creating a :Widget: named "a" yields a :Widget: with name "a". + - An empty name is rejected. + +- Delete a :Widget: by name. diff --git a/tests/fixtures/sample-crlf.plain b/tests/fixtures/sample-crlf.plain index e3f022b..389c64d 100644 --- a/tests/fixtures/sample-crlf.plain +++ b/tests/fixtures/sample-crlf.plain @@ -25,18 +25,11 @@ description: Fixture spec exercising every section. ***functional specs*** -- Create a :Widget: from a name. +# create-widget +A :Widget: is created from a name. - ***acceptance tests*** +# list-widgets +All :Widget: items in the :Registry: are listed. - > dropped comment inside the block - - Creating a :Widget: named "a" yields a :Widget: with name "a". - - An empty name is rejected. - -- List all :Widget: in the :Registry:. - - ***acceptance tests*** - - - An empty :Registry: lists nothing. - -- Delete a :Widget: by name. +# delete-widget +A :Widget: is deleted by name. diff --git a/tests/fixtures/sample.plain b/tests/fixtures/sample.plain index 2937f94..77081c9 100644 --- a/tests/fixtures/sample.plain +++ b/tests/fixtures/sample.plain @@ -25,18 +25,11 @@ description: Fixture spec exercising every section. ***functional specs*** -- Create a :Widget: from a name. +# create-widget +A :Widget: is created from a name. - ***acceptance tests*** +# list-widgets +All :Widget: items in the :Registry: are listed. - > dropped comment inside the block - - Creating a :Widget: named "a" yields a :Widget: with name "a". - - An empty name is rejected. - -- List all :Widget: in the :Registry:. - - ***acceptance tests*** - - - An empty :Registry: lists nothing. - -- Delete a :Widget: by name. +# delete-widget +A :Widget: is deleted by name. diff --git a/tests/fixtures/tests/sample/create-widget.plain b/tests/fixtures/tests/sample/create-widget.plain new file mode 100644 index 0000000..db18d7f --- /dev/null +++ b/tests/fixtures/tests/sample/create-widget.plain @@ -0,0 +1,19 @@ +--- +module: sample +func_id: create-widget +--- + +***conformance tests*** + +> a dropped comment inside the section + +# create-named-widget +Creating a :Widget: named "a" yields a :Widget: with name "a". + +# create-second-widget +A second :Widget: named "b" joins "a" in the :Registry:. + +***acceptance tests*** + +# reject-empty-name +An empty name is rejected and the :Registry: stays unchanged. diff --git a/tests/fixtures/tests/sample/list-widgets.plain b/tests/fixtures/tests/sample/list-widgets.plain new file mode 100644 index 0000000..2df422e --- /dev/null +++ b/tests/fixtures/tests/sample/list-widgets.plain @@ -0,0 +1,8 @@ +--- +module: sample +func_id: list-widgets +--- + +***conformance tests*** +> add conformance test scenarios here +***acceptance tests*** diff --git a/tests/run_plain_sections_tests.py b/tests/run_plain_sections_tests.py index 030d60e..a50949d 100644 --- a/tests/run_plain_sections_tests.py +++ b/tests/run_plain_sections_tests.py @@ -29,6 +29,9 @@ CRLF = os.path.join(FIXTURES, "sample-crlf.plain") SECOND = os.path.join(FIXTURES, "second.plain") EMPTY = os.path.join(FIXTURES, "no-sections.plain") +# The two test sections live in a test plain file, never in a spec plain file. +TEST_PLAIN = os.path.join(FIXTURES, "tests", "sample", "create-widget.plain") +OLD_SYNTAX = os.path.join(FIXTURES, "old-syntax.plain") harness.set_script(SCRIPT) @@ -48,20 +51,23 @@ def main(): def run_suite(tmp): # 1. Every section against its hand-authored expectation. - for sec in ("defs", "impl-reqs", "test-reqs", "func-specs", "acc-tests", "all"): + for sec in ("defs", "impl-reqs", "test-reqs", "func-specs", "all"): assert_stdout("section %s" % sec, expect(sec), sec, SAMPLE) + for sec in ("conf-tests", "acc-tests"): + assert_stdout("section %s" % sec, expect(sec), sec, TEST_PLAIN) # 2. Section aliases all resolve to the same body. for alias in ("definitions", "DEFS", ":plainDefinitions:", "Definitions"): assert_stdout("alias %s" % alias, expect("defs"), alias, SAMPLE) assert_stdout("alias impl", expect("impl-reqs"), "impl", SAMPLE) assert_stdout("alias implementation_reqs", expect("impl-reqs"), "implementation_reqs", SAMPLE) - assert_stdout('alias "acceptance tests"', expect("acc-tests"), "acceptance tests", SAMPLE) + assert_stdout('alias "acceptance tests"', expect("acc-tests"), "acceptance tests", TEST_PLAIN) + assert_stdout("alias :ConformanceTests:", expect("conf-tests"), ":ConformanceTests:", TEST_PLAIN) assert_stdout("alias everything", expect("all"), "everything", SAMPLE) # 3. CRLF input yields byte-identical output to the LF fixture. assert_stdout("crlf fixture, all sections", expect("all"), "all", CRLF) - assert_stdout("crlf fixture, acc-tests", expect("acc-tests"), "acc-tests", CRLF) + assert_stdout("crlf fixture, func-specs", expect("func-specs"), "func-specs", CRLF) # 4. --include-filename prefixes "## " and a blank line. want_filename = ("## %s\n\n" % SAMPLE).encode("utf-8") + expect("defs") @@ -137,6 +143,15 @@ def run_suite(tmp): else: bad("missing section warns on stderr, stdout empty") + # 9b. An indented ***acceptance tests*** block is not a section any more, so a spec in the + # old syntax yields no acceptance tests at all - test_scenarios.py is what names the cause. + assert_rc("nested acceptance tests are not a section", 0, "acc-tests", OLD_SYNTAX) + out, err = run("acc-tests", OLD_SYNTAX)[1:] + if out == b"" and err != b"": + ok("old-syntax spec warns, no acceptance tests body") + else: + bad("old-syntax spec warns, no acceptance tests body") + # 10. Flags are accepted after the positional arguments too. assert_stdout("flag after positionals", want_filename, "defs", SAMPLE, "--include-filename") diff --git a/tests/run_test_scenarios_tests.py b/tests/run_test_scenarios_tests.py new file mode 100644 index 0000000..5176096 --- /dev/null +++ b/tests/run_test_scenarios_tests.py @@ -0,0 +1,387 @@ +#!/usr/bin/env python3 +# Tests for skills/render-spec/scripts/test_scenarios.py. +# +# Usage: python3 tests/run_test_scenarios_tests.py (normally via tests/run_tests.py) +# Exit: 0 all assertions passed, 1 something failed. +# +# The assertion helpers live in tests/harness.py: byte-exact comparisons, every +# child process run with sys.executable, no shell anywhere. +# +# Most cases build a throwaway module in a temporary folder, because what is under test is +# how the script pairs a module spec with the test plain files next to it - a shape that a +# fixed fixture tree can only show once. + +import os +import shutil +import sys +import tempfile + +import harness +from harness import assert_rc, assert_stdout, bad, ok, read, run + +ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +SCRIPT = os.path.join(ROOT, "skills", "render-spec", "scripts", "test_scenarios.py") +FIXTURES = os.path.join(ROOT, "tests", "fixtures") +EXPECTED = os.path.join(ROOT, "tests", "expected") + +SAMPLE = os.path.join(FIXTURES, "sample.plain") +OLD_SYNTAX = os.path.join(FIXTURES, "old-syntax.plain") +EMPTY = os.path.join(FIXTURES, "no-sections.plain") + +harness.set_script(SCRIPT) + +SPEC = """--- +description: A module built for one assertion. +--- + +***functional specs*** + +# alpha +The first functionality. + +# beta +The second functionality. +""" + + +def write(path, text, newline="\n"): + folder = os.path.dirname(path) + if folder and not os.path.isdir(folder): + os.makedirs(folder) + with open(path, "wb") as handle: + handle.write(text.replace("\n", newline).encode("utf-8")) + return path + + +def module(tmp, name, spec=SPEC): + """A fresh folder holding `name`.plain, returned as the path of that spec.""" + folder = os.path.join(tmp, name) + return write(os.path.join(folder, name + ".plain"), spec) + + +def test_file(spec, module_name, func_id, body, name=None, newline="\n"): + """A test plain file under the spec's tests/ folder, at the conventional path.""" + folder = os.path.dirname(spec) + leaf = (name or func_id) + ".plain" + return write(os.path.join(folder, "tests", module_name, leaf), body, newline) + + +def main(): + tmp = tempfile.mkdtemp() + try: + run_suite(tmp) + finally: + shutil.rmtree(tmp, ignore_errors=True) + return harness.finish() + + +def run_suite(tmp): + # 1. The fixture tree end to end: one bound file, one empty file, one functionality with + # no file at all - and the warnings that name the last two. + assert_stdout("fixture module, grouped by func-id", + read(os.path.join(EXPECTED, "scenarios.txt")), SAMPLE) + err = run(SAMPLE)[2] + for want in (b"sample:list-widgets has no test scenario", + b"sample:delete-widget has no test plain file"): + if want in err: + ok("warns: %s" % want.decode("utf-8")) + else: + bad("warns: %s" % want.decode("utf-8"), err.decode("utf-8", "replace")) + assert_rc("a missing or empty test plain file still exits 0", 0, SAMPLE) + + # 2. Scenario bodies are copied verbatim, whatever shape they have. + spec = module(tmp, "shapes") + test_file(spec, "shapes", "alpha", """--- +module: shapes +func_id: alpha +--- + +***conformance tests*** + +> this comment is dropped + +# prose-scenario +One sentence that says what happens. + +# bullet-scenario +given: + - :App: is running +then: + - it answers + +***acceptance tests*** + +# rejects-nothing +Nothing is rejected. +""") + test_file(spec, "shapes", "beta", """--- +module: shapes +func_id: beta +--- + +***conformance tests*** +***acceptance tests*** +""") + want = b"""# Test scenarios of shapes + +> The complete list of tests to implement for this module. The renderer +> implements these scenarios and no others; it never invents one. + +## alpha + +> from tests/shapes/alpha.plain + +***conformance tests*** + +# prose-scenario +One sentence that says what happens. + +# bullet-scenario +given: + - :App: is running +then: + - it answers + +***acceptance tests*** + +# rejects-nothing +Nothing is rejected. + +## beta + +> from tests/shapes/beta.plain +> no test scenarios +""" + assert_stdout("scenario bodies are copied verbatim", want, spec) + + # 3. CRLF test plain files yield byte-identical output. + crlf = module(tmp, "crlf") + test_file(crlf, "crlf", "alpha", """--- +module: crlf +func_id: alpha +--- + +***conformance tests*** + +# one-scenario +It happens. + +***acceptance tests*** +""", newline="\r\n") + lf = module(tmp, "lf") + test_file(lf, "lf", "alpha", """--- +module: lf +func_id: alpha +--- + +***conformance tests*** + +# one-scenario +It happens. + +***acceptance tests*** +""") + if run(crlf)[1].replace(b"crlf", b"lf") == run(lf)[1]: + ok("crlf test plain file matches the lf one") + else: + bad("crlf test plain file matches the lf one") + + # 4. Two files bound to one functionality merge; a repeated scenario-id is skipped. + merged = module(tmp, "merged") + test_file(merged, "merged", "alpha", """--- +module: merged +func_id: alpha +--- + +***conformance tests*** + +# first +From the first file. + +***acceptance tests*** +""", name="a-first") + test_file(merged, "merged", "alpha", """--- +module: merged +func_id: alpha +--- + +***conformance tests*** + +# second +From the second file. + +# first +A repeat of a scenario-id already taken. + +***acceptance tests*** +""", name="b-second") + out, err = run(merged)[1:] + if b"# first\nFrom the first file." in out and b"# second\nFrom the second file." in out: + ok("two files bound to one functionality merge") + else: + bad("two files bound to one functionality merge", out.decode("utf-8", "replace")) + if b"A repeat of a scenario-id" not in out and b"scenario-id `first` twice" in err: + ok("a repeated scenario-id is skipped and warned about") + else: + bad("a repeated scenario-id is skipped and warned about") + + # 5. Files that do not belong to this module, or to no functionality of it. + strays = module(tmp, "strays") + test_file(strays, "strays", "alpha", """--- +module: strays +func_id: alpha +--- + +***conformance tests*** + +# bound +It is bound. + +***acceptance tests*** +""") + test_file(strays, "elsewhere", "alpha", """--- +module: elsewhere +func_id: alpha +--- + +***conformance tests*** + +# other-module +It belongs to another module. + +***acceptance tests*** +""") + write(os.path.join(os.path.dirname(strays), "tests", "notes.plain"), "no frontmatter here\n") + out, err = run(strays)[1:] + if b"other-module" not in out: + ok("another module's test plain file is left out") + else: + bad("another module's test plain file is left out") + if b"elsewhere" not in err: + ok("another module's test plain file is passed over in silence") + else: + bad("another module's test plain file is passed over in silence") + if b"has no `module:` and `func_id:`" in err: + ok("a .plain file with no frontmatter warns and is skipped") + else: + bad("a .plain file with no frontmatter warns and is skipped", err.decode("utf-8", "replace")) + assert_rc("a file of another module still exits 0", 0, strays) + + # 5b. A test plain file bound to no functionality stops the run: its scenarios were written + # for something - a renamed or deleted functionality, or a typo - and would be thrown away. + stale = module(tmp, "stale") + test_file(stale, "stale", "gamma", """--- +module: stale +func_id: gamma +--- + +***conformance tests*** + +# orphan +It names a functionality that is gone. + +***acceptance tests*** +""") + assert_rc("a test plain file bound to no functionality exits 1", 1, stale) + err = run(stale)[2] + if b"is not a functionality of stale" in err and b"tests/stale/gamma.plain" in err: + ok("the stale file and its func-id are named in the error") + else: + bad("the stale file and its func-id are named in the error", + err.decode("utf-8", "replace")) + + # 6. The syntax pyro no longer renders: a nested block, or bullets with no func-id. + assert_rc("a nested acceptance tests block exits 1", 1, OLD_SYNTAX) + if b"nested ***acceptance tests***" in run(OLD_SYNTAX)[2]: + ok("the nested block is named in the error") + else: + bad("the nested block is named in the error") + bullets = module(tmp, "bullets", """--- +description: Functional specs as bullets, the shape pyro no longer renders. +--- + +***functional specs*** + +- Create a :Widget: from a name. +""") + assert_rc("functional specs without a func-id heading exit 1", 1, bullets) + if b"no `# ` heading" in run(bullets)[2]: + ok("the missing heading is named in the error") + else: + bad("the missing heading is named in the error") + + # 7. Malformed ids. A func-id names files and render steps, so it stops the run; a + # scenario-id only labels a test, so it warns and the scenario is still rendered. + shouty = module(tmp, "shouty", SPEC.replace("# alpha", "# Alpha Spec")) + assert_rc("a func-id that is not lowercase-hyphen exits 1", 1, shouty) + loud = module(tmp, "loud") + test_file(loud, "loud", "alpha", """--- +module: loud +func_id: alpha +--- + +***conformance tests*** + +# Shouty Scenario +It still renders. + +***acceptance tests*** +""") + out, err = run(loud)[1:] + if b"is not a scenario-id" in err and b"# Shouty Scenario" in out: + ok("a malformed scenario-id warns and the scenario is kept") + else: + bad("a malformed scenario-id warns and the scenario is kept") + + # 8. A module with no functional specs at all is a warning, not a failure. + assert_rc("a spec with no functional specs exits 0", 0, EMPTY) + if b"no functional specs" in run(EMPTY)[2]: + ok("the empty spec warns on stderr") + else: + bad("the empty spec warns on stderr") + + # 9. --output writes exactly what stdout would carry, UTF-8 with LF, parents created. + via_stdout = run(SAMPLE)[1] + via_output = os.path.join(tmp, "out", "scenarios.md") + assert_rc("--output creates missing parent folders", 0, "--output", via_output, SAMPLE) + if read(via_output) == via_stdout: + ok("--output matches stdout byte for byte") + else: + bad("--output matches stdout byte for byte") + if b"\r" not in read(via_output): + ok("--output keeps LF endings") + else: + bad("--output keeps LF endings") + utf8 = module(tmp, "utf8") + test_file(utf8, "utf8", "alpha", """--- +module: utf8 +func_id: alpha +--- + +***conformance tests*** + +# naive-widget +A :Widget: named "Widžet" is created - naïve. + +***acceptance tests*** +""") + utf8_out = os.path.join(tmp, "utf8.md") + assert_rc("--output= exits 0", 0, "--output=" + utf8_out, utf8) + if "Widžet".encode("utf-8") in read(utf8_out): + ok("--output keeps non-ASCII text UTF-8") + else: + bad("--output keeps non-ASCII text UTF-8") + assert_rc("--output to unwritable path exits 1", 1, + "--output", os.path.join(via_output, "y"), SAMPLE) + + # 10. Unreadable inputs exit 1, usage errors exit 2. + assert_rc("missing spec exits 1", 1, os.path.join(FIXTURES, "does-not-exist.plain")) + assert_rc("directory argument exits 1", 1, FIXTURES) + assert_rc("no arguments exits 2", 2) + assert_rc("two specs exit 2", 2, SAMPLE, EMPTY) + assert_rc("--output without a path exits 2", 2, SAMPLE, "--output") + assert_rc("--help exits 2", 2, "--help") + + +if __name__ == "__main__": + sys.exit(main())