Repository navigation
Conversation
Universal-Commerce-Protocol#120 replaced the CI loop over `*_test.py` with `ucp-conformance --suite=all`, which resolves through `SUITE_REGISTRY`. A module the registry does not name contributes no cases to that run, so it merges and never executes. The registry already reports a name that does not resolve: injecting a typo gives `ERROR: Unknown test specifier` and exit 1. It says nothing about the reverse. Adding an unregistered module leaves `--suite=all` at its previous count, exit 0, with the module never mentioned. `unregistered_modules()` discovers `*_test.py` by dotted path relative to the repository root, excluding the `framework` harness package, and diffs against the union of `SUITE_REGISTRY`. `--verify-registry` lists anything unreachable and exits non-zero. A CI step runs the unit tests and the check after the dependency sync and before the server boots, since neither contacts anything. Discovery is by dotted path so the check keeps working after the tiered layout lands, where it reports `shopping.checkout.protocol_test` rather than a bare file name. Eight unit tests cover the check. Seven mutations each red at least one: returning an empty list, dropping the harness exclusion, ignoring the registry, searching one level instead of recursing, keeping the `.py` suffix, dropping the hidden-directory exclusion, and returning 0 from `--verify-registry` while offenders are reported.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this adds
--verify-registry, which fails when a test module in the tree is not named bySUITE_REGISTRY, and a CI step that runs it.Why
#120 replaced the workflow loop over
*_test.pywithucp-conformance --suite=all, whichresolves through
SUITE_REGISTRY. A module the registry does not name contributes no cases tothat run, so it merges and never executes.
The registry already reports the opposite mistake. Injecting a name that does not resolve:
It says nothing about a module it does not name. Adding an unregistered module to the tree:
The count does not move and the module is never mentioned. So the registry is loud about a
name that resolves to nothing and silent about a module that nothing names.
This is reachable today rather than hypothetical. Four open pull requests add a
*_test.pywithout an entry: #89, #108 and #112, which are mine and which I have just corrected, and #118,
which still has none. #112 carries two approvals, so without its entry it would have merged
with three cases that never ran. #125 and the #121 series do register their modules.
How it works
unregistered_modules()discovers*_test.pyunder the repository root and compares the setagainst the union of
SUITE_REGISTRY. Discovery is by dotted path, so a tiered layout reportsshopping.checkout.protocol_testrather than a bare file name. Theframeworkpackage isexcluded:
base_test.pythere is the shared base class, not a suite module, and it is the only*_test.pyin that package. Hidden directories are excluded so a virtualenv that ships testmodules cannot produce false reports.
It survives the restructure. Cherry-picked onto #121 and onto #122, the commit applies with no
conflict, the unit tests pass, and
--verify-registryreports OK against their dotted-pathregistries, which use the
commonandcorekeys.The CI step
Placed after the dependency sync and before the server boots, because neither the unit tests
nor the check contacts anything, so this fails fast and needs no fixture.
It is not marked
continue-on-error, unlike the conformance test step beside it. That steptolerates failure on pull requests because it depends on a live server. This one is a static
comparison of two sets in the repository and is deterministic, so there is nothing for it to be
tolerant of. Worth noting that the required status checks on this branch are
cla/googleandGovernance / Approvals, so neither step gates merging either way; this changes the colour ofthe job, not whether a pull request can land.
Verification
Run on this branch, exit codes read from the process:
python -m unittest test_suite_registryucp-conformance --verify-registryucp-conformance --suite=all --dry-runpre-commit run --all-filesSeven mutations, each reverted and the cache cleared between runs, each reds at least one test:
returning an empty list reds 2, dropping the
frameworkexclusion reds 2, ignoring the registryreds 3, searching one level instead of recursing reds 1, keeping the
.pysuffix reds 4,dropping the hidden-directory exclusion reds 1, and returning 0 from
--verify-registrywhileoffenders are reported reds 1. That last one is the reason the exit code has its own test: the
CI step depends on it, and without the test the check could report offenders and still pass.
The unit file is named
test_suite_registry.pyrather than*_test.pyso the check does notdiscover its own test file, and a test pins that.