Skip to content

e2e-testing.md

Latest commit

 

History

History
160 lines (121 loc) · 8.09 KB

e2e-testing.md

File metadata and controls

160 lines (121 loc) · 8.09 KB
Raw

E2E/Integration Testing

The Solr Operator "unit" tests cannot fully test the Solr Operator, they can only test interactions with the Kubernetes API Server. Because of this, many features are untestable because they require communicating with a running SolrCloud cluster.

The E2E (End-to-end), or integration, test suite enables the project to fully test the operator on a real "KiND" Kubernetes cluster. Thus every feature can be fully tested.

How to run the tests

There is an easy make target to run the e2e tests with default parameters.

$ make e2e-tests

This will create a KinD Cluster to act as the Kubernetes cluster, and install the Solr Operator via its Helm Chart.

It will run the e2e-tests using parallel test-runners, and each test runner will use its own namespace. The Solr Operator will be deployed to listen on all namespaces.

If all tests succeed, then the KinD Cluster will be deleted. Please see below for information on test failures.

The following sections describe all the ways for you to customize the e2e tests.

Customizing the test parameters

The Solr Operator integration tests are meant to be customizable to test the operator in a range of use cases. Currently you have to use a KinD cluster, but you can still change the Kubernetes Version and Solr Image to test with. Future iterations will hopefully enable testing with existing clusters.

Beyond changing the Solr Image and Kubernetes environment, the test suite itself can be customized for parallelization and randomization.

Example:

$ make e2e-tests TEST_SEED=89724023 SOLR_IMAGE=apache/solr-nightly:10.0.0-SNAPSHOT KUBERNETES_VERSION=v1.26.4

Options

  • TEST_SEED - Equivalent to Ginkgo's --seed. If set, randomization in the test framework will be seeded with this number.
  • TEST_PARALLELISM - Equivalent to Ginkgo's --procs. Ginkgo will use this many parallel test runners. The default parallelism is 3.
  • SOLR_IMAGE - The solr docker image label to use in the integration tests. It is recommended to use only supported versions for the Solr Operator version being tested. Default is solr:9.11.0.
  • KUBERETES_VERSION - A full Kubernetes version, starting with v, to use when creating the KinD Cluster. To find a list of all possible versions, check the KinD Node Docker tags. Default is v1.26.6.

Filtering tests

The full test suite might take too long if you just want to test a specific feature. There are a number of ways to filter the tests that are run, and each can be specified via an environment variable.

Options

  • TEST_FILES - Equivalent to Ginkgo's --focus-file. If set, tests will only run specs in matching files. Accepts: [file (regexp) | file:line | file:lineA-lineB | file:line,line,line]
  • TEST_FILTER - Equivalent to Ginkgo's --focus. If set, only specs that match this regular expression will be run.
    NOTE: The spec is the concatenation of all levels of hierarchy that an It() test belongs in. See below for more information.
  • TEST_LABELS - Equivalent to Ginkgo's --label-filter. If set, only specs with labels that match the label-filter will be run. Note: A test has to have labels for this to be effective. The only label in use today is pr, which selects the reduced profile run in CI.
    The passed-in expression can include boolean operations (!, &&, ||, ,), groupings via (), and regular expressions /regexp/. e.g. (cat || dog) && !fruit
  • TEST_SKIP - Equivalent to Ginkgo's --skip. Do not run tests that match this regular expression string. NOTE: The same rules on spec names from TEST_FILTER apply here.

Test/Spec name matching

$ make e2e-tests TEST_FILTER="E2E - Backups Local Directory - Recurring Takes a backup correctly"

The above test's hierarchy is:

  • Describe("E2E - Backups"
  • Context("Local Directory - Recurring"
  • It("Takes a backup correctly"

Thus we concatenate all 3 together, with each name separated by a space, to get the unique test filter.

If TEST_FILTER="E2E - Backups Local Directory - Recurring" is used, then all tests under the first two test Describe and Context will be run.

Since this is a regex string, using just TEST_FILTER="Backups" will match any Describe, Context, or It that contains the work "Backups".

Customizing the test environment

It is important to have control over the KinD cluster that is created/used when running the e2e tests. The following options are aimed at opening this up.

Options

  • REUSE_KIND_CLUSTER_IF_EXISTS - Defaults to true. If a kind cluster for the exact same setup (solr image, kube version, operator version), already exists, this option determines whether to use that cluster for the tests or to delete and recreate the cluster before tests are run.
  • LEAVE_KIND_CLUSTER_ON_SUCCESS - Defaults to false. On test failures, the KinD cluster is not deleted afterwards, so that tests can be quickly rerun with the same environment. If this option is set to true, then the KinD cluster will not be deleted afterwards, even if all tests succeed. This might be useful when quickly iterating on tests that succeed, to reduce the time to create new clusters.

Test Failures

If a test fails, Ginkgo will print out a command that will retest the individual test that failed. If multiple tests in a run fail, then Ginkgo will print out a command for each failed test. This command will also rerun the test with the same randomization seed that the test failed with.

If a test fails, the KinD Kubernetes Cluster will not be deleted. This way the tests can be re-run using the same cluster that they failed on previously. Use the environment variable REUSE_KIND_CLUSTER_IF_EXISTS=false to re-create the KinD cluster, instead of reusing the existing one.

Debugging output for each spec (pod logs, events, resource status and the operator log) is written to tests/e2e/output/. Ginkgo also writes junit.xml and report.json to that directory, containing the failure messages of the whole run. Since the directory is recreated at the start of every run, copy these files elsewhere if you need to keep them.

Continuous Integration

The full suite is too heavy for the GitHub-hosted runners available to the project (4 vCPU and 16 GB of memory), so pull requests do not run all of it. Instead, the e2e workflow runs the pr profile: the specs that carry the Ginkgo label pr, selected with TEST_LABELS=pr. The profile holds one cheap, representative spec per feature area (basic startup, security.json, TLS via secrets and via a mounted directory, scale up and scale down with replica migration, backups, the Prometheus exporter, persistent storage expansion and ingress/addressability). Slow specs, such as the rolling upgrade, and the many variants of the same feature only run in the full suite.

The workflow only runs for pull requests that touch something the suite exercises: the Go code and the operator image, the CRDs and dependencies installed into the test cluster, the solr-operator Helm chart, and the e2e harness itself. Changes to documentation, examples, release tooling or the solr Helm chart do not trigger it.

The workflow can also be started manually from the GitHub Actions UI. Leave the label filter empty to run the full suite, or pick another Solr or Kubernetes version.

To run exactly what CI runs:

$ make e2e-tests TEST_LABELS=pr TEST_PARALLELISM=2

When adding a spec, give it the pr label only if it exercises an operator feature the profile does not cover yet, and prefer specs that need at most three Solr pods. Labels go on the Context (or It) as a decorator, e.g. FContext("Provided Zookeeper", Label("pr"), func() {.

IntelliJ & GoLand Testing

The e2e-tests can be tested via IDEA, just as the unit tests can, though creating the cluster will make the tests significantly slower. Please refer to the IDEA Testing Docs for more information.