6. Automated Testing

The UFS Weather Model repository on GitHub employs two types of automated testing:

  1. CI/CD (Continuous Integration/Continuous Development) testing on the cloud

  2. AutoRT on NOAA R&D platforms

Both are application level tests and utilize the regression testing framework discussed in Section 3.5.1.

6.1. CI/CD

The UFS Weather Model (WM) uses GitHub Actions (GHA), a GitHub-hosted continuous integration service, to perform CI/CD testing. Build jobs are done on GHA-provided virtual machines. Test jobs are performed on the Amazon Web Services (AWS) cloud platform using a number of EC2 instances. Builds and tests are carried out in a Docker container. The container includes a pre-installed version of the HPC-Stack, which includes all prerequisite libraries. Input data needed to run the tests are stored as a separate Docker container.

When a developer makes a pull request (PR) to the UFS WM repository, a code manager may add the run-ci label, which triggers the CI/CD workflow. The CI/CD workflow then executes the following steps:

  1. A check is performed to make sure the UFS Weather Model and its first level subcomponents are up to date with the top of the develop branch.

  2. If the check is successful, build jobs are started on GHA-provided virtual machines by downloading the hpc-stack Docker container stored in Docker Hub.

  3. Once all build jobs are successful, the created executable files are stored as artifacts in GHA.

  4. A number of AWS EC2 instances are started.

  5. Test jobs are started on AWS after downloading the hpc-stack Docker container, the executable file from the build job, and the input-data Docker container.

  6. When all tests are complete, EC2 instances are stopped. Test results are reported on GitHub.

The GHA-related yaml scripts are located in the .github/workflows/ directory. build_test.yml is the main workflow file, and aux.yml is an auxiliary file responsible for (1) checking that the PR branch is up-to-date and (2) starting/stopping the EC2 instances.

Other CI-related scrips are located in the tests/ci/ directory. ci.sh is the main script that invokes Docker build and run. Dockerfile is used to build the UFS Weather Model. Other shell and python scripts help with various tasks. For example:

  • repo_check.sh checks that the PR branch is up-to-date.

  • check_status.py checks the status of EC2 instances.

  • setup.py and ci.test configure the test cases to execute in the CI/CD workflow.

6.2. Auto RT

The Automated Regression Testing (AutoRT) system is a python program that automates the process of regression testing on NOAA HPC platforms. It contains the files in Table 6.1 below:

Table 6.1 Files for Automated Regression Testing (AutoRT) system

File Name

Description

start_rt_auto.sh

Verifies HPC name, sets the python paths

rt_auto.py

Python interface between the HPC and the github API

jobs/bl.py

Functions for the baseline job

jobs/rt.py

Functions for the regression test job

6.2.1. AutoRT Workflow

On supported HPC systems, a cron job runs the start_rt_auto.sh bash script every 15 minutes. This script checks the HPC name and sets certain python paths. Then, it runs rt_auto.py, which uses the Github API (through pyGitHub) to check the labels on pull requests to ufs-weather-model. If a PR label matches the HPC name (e.g., hera-intel-RT or cheyenne-gnu-BL), the label provides the HPC with the compiler and job information to run a test or task on the machine. If no PR label matches HPC name, the script exits.

For example, a PR labeled gaea-intel-BL will be recognized by the HPC machine ‘Gaea’. It will set the RT_COMPILER variable to ‘intel’ and run the baseline creation script (bl.py). This script creats a job class that contains all information from the machine that the job will need to run. That information is sent into the jobs/rt[bl].py script.

rt.py sets directories for storage, gets repo information, runs the regression test, and completes any required post processing.

def run(job_obj):
   logger = logging.getLogger('RT/RUN')
   workdir = set_directories(job_obj)
   branch, pr_repo_loc, repo_dir_str = clone_pr_repo(job_obj, workdir)
   run_regression_test(job_obj, pr_repo_loc)
   post_process(job_obj, pr_repo_loc, repo_dir_str, branch)

bl.py: (similar to rt.py) Adds functionality to create baselines before running regression testing.

   def run(job_obj):
      logger = logging.getLogger('BL/RUN')
      workdir, rtbldir, blstore = set_directories(job_obj)
      pr_repo_loc, repo_dir_str = clone_pr_repo(job_obj, workdir)
      bldate = get_bl_date(job_obj, pr_repo_loc)
      bldir = f'{blstore}/develop-{bldate}/{job_obj.compiler.upper()}'
      bldirbool = check_for_bl_dir(bldir, job_obj)
      run_regression_test(job_obj, pr_repo_loc)
      post_process(job_obj, pr_repo_loc, repo_dir_str, rtbldir, bldir)