Skip to content

feat(runner): fail when a test module is not named by SUITE_REGISTRY - #127

Open
vishkaty wants to merge 1 commit into
Universal-Commerce-Protocol:mainfrom
vishkaty:feat/verify-suite-registry
Open

vishkaty wants to merge 1 commit into
Universal-Commerce-Protocol:mainfrom
vishkaty:feat/verify-suite-registry

Conversation

@vishkaty

Copy link
Copy Markdown
Contributor

What this adds

--verify-registry, which fails when a test module in the tree is not named by
SUITE_REGISTRY, and a CI step that runs it.

Why

#120 replaced the workflow loop over *_test.py with ucp-conformance --suite=all, which
resolves through SUITE_REGISTRY. A module the registry does not name contributes no cases to
that run, so it merges and never executes.

The registry already reports the opposite mistake. Injecting a name that does not resolve:

$ ucp-conformance --suite=all --dry-run
ERROR: Unknown test specifier: ap2_tset
Error: could not load test specifier(s): ap2_tset
$ echo $?
1

It says nothing about a module it does not name. Adding an unregistered module to the tree:

$ ucp-conformance --suite=all --dry-run | grep -c '^  - '
78
$ echo $?
0

The count does not move and the module is never mentioned. So the registry is loud about a
name that resolves to nothing and silent about a module that nothing names.

This is reachable today rather than hypothetical. Four open pull requests add a *_test.py
without an entry: #89, #108 and #112, which are mine and which I have just corrected, and #118,
which still has none. #112 carries two approvals, so without its entry it would have merged
with three cases that never ran. #125 and the #121 series do register their modules.

How it works

unregistered_modules() discovers *_test.py under the repository root and compares the set
against the union of SUITE_REGISTRY. Discovery is by dotted path, so a tiered layout reports
shopping.checkout.protocol_test rather than a bare file name. The framework package is
excluded: base_test.py there is the shared base class, not a suite module, and it is the only
*_test.py in that package. Hidden directories are excluded so a virtualenv that ships test
modules cannot produce false reports.

It survives the restructure. Cherry-picked onto #121 and onto #122, the commit applies with no
conflict, the unit tests pass, and --verify-registry reports OK against their dotted-path
registries, which use the common and core keys.

The CI step

Placed after the dependency sync and before the server boots, because neither the unit tests
nor the check contacts anything, so this fails fast and needs no fixture.

It is not marked continue-on-error, unlike the conformance test step beside it. That step
tolerates failure on pull requests because it depends on a live server. This one is a static
comparison of two sets in the repository and is deterministic, so there is nothing for it to be
tolerant of. Worth noting that the required status checks on this branch are cla/google and
Governance / Approvals, so neither step gates merging either way; this changes the colour of
the job, not whether a pull request can land.

Verification

Run on this branch, exit codes read from the process:

gate result
python -m unittest test_suite_registry 8 passed
ucp-conformance --verify-registry OK, exit 0
ucp-conformance --suite=all --dry-run 78 cases, unchanged from main
pinned pre-commit run --all-files 10 hooks passed

Seven mutations, each reverted and the cache cleared between runs, each reds at least one test:
returning an empty list reds 2, dropping the framework exclusion reds 2, ignoring the registry
reds 3, searching one level instead of recursing reds 1, keeping the .py suffix reds 4,
dropping the hidden-directory exclusion reds 1, and returning 0 from --verify-registry while
offenders are reported reds 1. That last one is the reason the exit code has its own test: the
CI step depends on it, and without the test the check could report offenders and still pass.

The unit file is named test_suite_registry.py rather than *_test.py so the check does not
discover its own test file, and a test pins that.

Universal-Commerce-Protocol#120 replaced the CI loop over `*_test.py` with `ucp-conformance --suite=all`,
which resolves through `SUITE_REGISTRY`. A module the registry does not name
contributes no cases to that run, so it merges and never executes.

The registry already reports a name that does not resolve: injecting a typo
gives `ERROR: Unknown test specifier` and exit 1. It says nothing about the
reverse. Adding an unregistered module leaves `--suite=all` at its previous
count, exit 0, with the module never mentioned.

`unregistered_modules()` discovers `*_test.py` by dotted path relative to the
repository root, excluding the `framework` harness package, and diffs against
the union of `SUITE_REGISTRY`. `--verify-registry` lists anything unreachable
and exits non-zero. A CI step runs the unit tests and the check after the
dependency sync and before the server boots, since neither contacts anything.

Discovery is by dotted path so the check keeps working after the tiered
layout lands, where it reports `shopping.checkout.protocol_test` rather than
a bare file name.

Eight unit tests cover the check. Seven mutations each red at least one:
returning an empty list, dropping the harness exclusion, ignoring the
registry, searching one level instead of recursing, keeping the `.py` suffix,
dropping the hidden-directory exclusion, and returning 0 from
`--verify-registry` while offenders are reported.
@damaz91 damaz91 added the status:needs-triage Signal that the PR is ready for human triage label Nov 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

status:needs-triage Signal that the PR is ready for human triage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants