GitHub Actions CI: CPU compilers with ECT validation

[cenamiller]

This PR adds some initial GitHub Actions continuous integration (CI) for MPAS-A. These workflows do automated build and correctness testing for Intel OneAPI, NVHPC, and GNU compilers on CPU.

This is a subset of the testing that's been developed on the
github.com/NCAR/MPAS-Model-CI fork.

The goal of this testing is to catch build issues and validate that
the model output remains within the bounds of internal variability
using the Ensemble Consistency Test
(ECT/PyCECT; Price-Broncucia et al. 2025,
doi:10.5194/gmd-18-2349-2025).

These workflows:

• Build MPAS-A in double precision with SMIOL I/O inside
  NCAR hpcdev Docker containers on GitHub-hosted runners.
  (PIO testing is available on the MPAS-Model-CI fork and
  will be included in future PRs)
• Test three compiler families (GNU, Intel OneAPI, NVHPC)
  with MPICH. (OpenMPI testing is available on the
  MPAS-Model-CI fork and will be included in future PRs)
  All run on new PRs against develop (or when someone pushes
  a new commit to an open PR) and on push to master.
  Only the GNU + MPICH test also runs on push to develop.
• Validate correctness by running 3 perturbed ensemble
  members and comparing against a previously generated
  200-member PyCECT ensemble summary

README.md was reverted to base. CI status badges can be added
as a future feature.

Test case data and ECT ensemble summaries are still hosted as
GitHub release assets on NCAR/MPAS-Model-CI. The
DATA_REPOSITORY variable in .github/ci-config.env
controls where data is downloaded from.

File descriptions:

• .github/ci-config.env — central CI configuration
• .github/actions/ — 9 composite actions (build-mpas,
  download-testdata, resolve-container, run-mpas,
  run-perturb-mpas, validate-ect, mpas-version,
  print-mpas-logs, ect-summary)
• .github/workflows/_test-compiler.yml — reusable CPU
  build + ECT validation workflow
• .github/workflows/test-{gcc,intel,nvhpc}-mpich.yml —
  3 caller workflows
• .github/data/ect_excluded_vars.txt — ECT variable
  exclusion list

Developed in NCAR/MPAS-Model-CI. Claude assistance.

[cenamiller]

I think the tests will be able to start running against 'develop' PRs once these workflows are on the 'develop' branch, and we shouldn't need to add everything to the master branch yet like I told you I was afraid we had to do. (Caveat: parts of the github actions UI might not show up until the workflows are on the default branch, but I think we can still do testing and make sure things are working how they should without that until the next release)

*pull_request and push aren't listed as needing to be on the default branch, just the merge ref https://docs.github.com/en/actions/reference/workflows-and-actions/events-that-trigger-workflows

[abishekg7]

Thanks for this work, @cenamiller! It will be a much needed addition to the MPAS dev workflow. Here are my suggestions so far - but let's check with @mgduda first before going ahead with proposed changes.

• We probably need to test the builds of the MPAS init_atmosphere CORE with at least one toolchain. Let's hold off on run tests for the init_atmosphere CORE for now.

• The number of tests that launch by default on every push can be reduced. By default, just building CORE=atmosphere with GNU and MPICH might be enough, and perhaps the ECT runs following that too. The rest of the test suite could be manually triggered via workflow_dispatch probably.

  But I was also wondering how simple it might be to trigger by issue_comment. It would be nice to have the ability to launch targeted tests, without having to run all other tests in the suite. I can say more about this offline.

• In the reporting screen, at the bottom of a PR, I see about 24 checks being performed. For each test case, there is config, build, ect validate, cleanup. I was wondering if we could just have one line item for each test case, and not separately list config and build, etc. It's fine for now, but wondering if it might be a long list when we add more test cases.

• Re. .github/workflows/test-compiler-mpi.yml, is there an advantage to keep these as separate files, or would it be okay to fold them into a matrix inside _test-compiler.yml

• Eventually, it would be good to have GPU tests and tests for idealized case, etc. But it might be better to postpone them to a next iteration of the CI.

• I was wondering how difficult adding a CMake build test here would be. That could benefit some developers downstream.

[cenamiller]

Hi @abishekg7, thank you for the comments!

• I agree on a small test to build the init_atmosphere core. I'll create and test it over in ncar/mpas-model-ci, and have it ready for the next iteration of CI if that's okay. It will use the existing framework
• Just to clarify, the 6 tests should only run on pushes/prs to the master and develop branches, or by manual dispatch. So they shouldn't be running on every push. Do we still think that's too much? I do think an nvhpc build is an important check
• "In the reporting screen, at the bottom of a PR, I see about 24 checks being performed. For each test case, there is config, build, ect validate, cleanup. I was wondering if we could just have one line item for each test case, and not separately list config and build, etc. It's fine for now, but wondering if it might be a long list when we add more test cases." > Yeah I'm not a fan of that either. This isn't an ultimate solution, but if you go to the 'checks' tab at the top of the PR, it groups them by test.
• The reason/advantage is that each yaml file gets its own badge and can individually report a failure. But it's a matter of preference. The reusable actions support moving back to a matrix approach
• agreed, GPU testing is implemented in NCAR/MPAS-Model-CI, but we need to address some security concerns with the main repo before we move it over
• Yes, a cmake build test is also on the list to do next :) I recommend we try it in NCAR/MPAS-Model-CI and then move it over

[cenamiller]

@abishekg7 Hi again, I addressed some of your other concerns with 8c35e70

• Missing-data guidance now points to the ECT generation workflow in NCAR/MPAS-Model-CI.
• /container/config_env.sh is now optional, and the NVHPC Makefile edits fail clearly if the expected lines cannot be found.
• I removed the unused run-mpas action from this PR. It can return with the later workflows that use it, good call
• About the python3 -c indentation concern... It turns out YAML "removes the common block indentation before passing the script to bash." So that's why it doesn't cause issues (and hasn't been an issue during testing on NCAR/MPAS-Model-CI)

[cenamiller]

Hi @abishekg7 and @mgduda, I've reduced the number/frequency of tests this initial CI PR contains like we discussed. The tests are more limited in scope now, just a CPU build and ensemble consistency test with GNU, Intel OneAPI, and NVHPC. Only MPICH is used, but OpenMPI remains an easy option for the future. The events that would trigger the tests are opening a PR against 'develop' or when there's a push on 'master' (like a new release). The tests can be triggered for old PRs by pushing a new commit to them, or by closing and reopening the PR. I also removed the CI badges section of the README for now, but it will be easy to add back once we decide which tests we want to have reporting their status on the README.

[mgduda]

@cenamiller I had merged this PR locally to develop, and accidentally pushed to the wrong remote (i.e., to MPAS-Dev/MPAS-Model), which closed this PR. I've reset the HEAD of the develop branch, but I think GitHub will forever consider this PR merged. Could you open a new PR with these changes?

[mgduda]

This PR has been replaced by PR #1524.