Repository navigation
tests: consistent test names and group suites - #4221
Merged
Merged
Conversation
Half of the tests of the nvme binary end in -cli, the other half do not, and some names use underscores or a space. Name each test after the feature it covers, in kebab-case like the other tests: nvme-cli - print-cli -> nvme-cli - print nvme-cli - fabrics-mock-cli -> nvme-cli - fabrics nvme-cli - argconfig_parse -> nvme-cli - argconfig-parse nvme-cli - plugin dispatch -> nvme-cli - plugin-dispatch The test of the e2e runner is no e2e test, but its name starts like one. Name it after the runner, nvme-cli-e2e - tap. Signed-off-by: Daniel Wagner <dwagner@suse.de>
The plugin tests run as part of nvme-cli and repeat the plugin name in the test name and in two suites: micron+micron-cli - nvme-cli:nvme-cli - micron_latency_mock_test Use the plugin as the component, as the e2e plugin tests do, and name the test after the command it covers, without the _mock_test suffix: nvme-cli:micron - latency The micron and ocp names are derived from the file names, so the files keep their names. Drop the plugin suites, nothing uses them: a name pattern selects the tests of a plugin, e.g. meson test 'micron - *'. Signed-off-by: Daniel Wagner <dwagner@suse.de>
The fabrics-conf target fuzzes the libnvme config parser, but its test is named after nvme-cli. Let a target name its component, and name the fabrics-conf test libnvme - fuzz-fabrics-conf. Signed-off-by: Daniel Wagner <dwagner@suse.de>
Some daemon test names contain spaces or the option they test, e.g. 'nvme-keysd - missing --config'. Use kebab-case like the other tests: nvme-keysd - --help -> nvme-keysd - help nvme-keysd - unknown option -> nvme-keysd - unknown-option nvme-keysd - missing --config -> nvme-keysd - missing-config nvme-keysd - kmip tls -> nvme-keysd - kmip-tls Signed-off-by: Daniel Wagner <dwagner@suse.de>
The NBFT tests are named after their table file, e.g. libnvme - NBFT-rhpoc, the Python binding tests start with python-. Group them with a / like the header tests: libnvme - nbft/rhpoc and libnvme - python/config. Signed-off-by: Daniel Wagner <dwagner@suse.de>
The e2e test names are the file names, e.g. nvme-cli - nvme_id_ctrl_test, and the plugin e2e tests are in a suite of their plugin. A mock test and an e2e test of the same command cannot be told apart by their name once the file name suffixes are gone. Name the e2e tests e2e-<command>, e.g. nvme-cli - e2e-id-ctrl and micron - e2e-id-ctrl, and put all of them in the e2e suite. Update the README of the plugin e2e tests to select them with it. Signed-off-by: Daniel Wagner <dwagner@suse.de>
Only the fuzz and e2e tests are in a suite, so there is no way to run,
e.g., just the fast tests which need neither a device nor a mock. Put
every test in exactly one group, which says how the test runs:
unit a test program for library or internal code
mock the nvme binary or a daemon, with a mock where it needs a
device
e2e the nvme binary against a real device
fuzz a fuzz target over its corpus
check a consistency check of the tree: headers, symbols, docs,
install tags, generated completions
meson test --suite unit --suite mock then runs all behaviour tests,
and meson test prints the group in front of every test.
Signed-off-by: Daniel Wagner <dwagner@suse.de>
Describe how tests are named, what the groups mean and how to select tests by group or by name. Signed-off-by: Daniel Wagner <dwagner@suse.de>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The test names are not consistent: some end in
-cli, some in_mock_test, some use underscores or spaces, and the plugin tests repeat the plugin name up to three times:This series applies one rule: every test is named
<component> - <test>and is in exactly one group suite.<component>is what the test covers:nvme-cli,libnvme,shared, a daemon or a plugin.<test>is kebab-case and names the feature, without-cli/_mock_testsuffixes. Fuzz and e2e tests start withfuzz-/e2e-, so they don't collide with the unit and mock tests of the same thing. Tests over a set of inputs use/like the existingheader/tests:nbft/,python/.unitmocke2efuzzcheckmeson test --suite unit --suite mockruns all behaviour tests, and a name pattern selects a component, e.g.meson test 'micron - *'. The unused plugin suites (micron-cli, ...) are dropped;--suite fuzzworks as documented before.Documentation/TESTING.mdgets a section on the names and groups.Nothing in CI or scripts refers to the old names. Each commit configures on its own (229 tests, no duplicate names, e2e names checked with
-De2e-tests=true), and the test suite passes.