6. Automated Testing¶
The UFS Weather Model repository on GitHub employs two types of automated testing:
CI/CD (Continuous Integration/Continuous Development) testing on the cloud
AutoRT on NOAA R&D platforms
Both are application level tests and utilize the regression testing framework discussed in Section 3.5.1.
6.1. CI/CD¶
The UFS Weather Model (WM) uses GitHub Actions (GHA), a GitHub-hosted continuous integration service, to perform CI/CD testing. Build jobs are done on GHA-provided virtual machines. Test jobs are performed on the Amazon Web Services (AWS) cloud platform using a number of EC2 instances. Builds and tests are carried out in a Docker container. The container includes a pre-installed version of the HPC-Stack, which includes all prerequisite libraries. Input data needed to run the tests are stored as a separate Docker container.
When a developer makes a pull request (PR) to the UFS WM repository, a code manager may add the run-ci label, which triggers the CI/CD workflow. The CI/CD workflow then executes the following steps:
A check is performed to make sure the UFS Weather Model and its first level subcomponents are up to date with the top of the
develop
branch.If the check is successful, build jobs are started on GHA-provided virtual machines by downloading the hpc-stack Docker container stored in Docker Hub.
Once all build jobs are successful, the created executable files are stored as artifacts in GHA.
A number of AWS EC2 instances are started.
Test jobs are started on AWS after downloading the hpc-stack Docker container, the executable file from the build job, and the input-data Docker container.
When all tests are complete, EC2 instances are stopped. Test results are reported on GitHub.
The GHA-related yaml
scripts are located in the .github/workflows/
directory.
build_test.yml
is the main workflow file, and aux.yml
is an auxiliary
file responsible for (1) checking that the PR branch is up-to-date and
(2) starting/stopping the EC2 instances.
Other CI-related scrips are located in the tests/ci/
directory. ci.sh
is the main script that
invokes Docker build and run. Dockerfile
is used to build the UFS Weather Model.
Other shell and python scripts help with various tasks. For example:
repo_check.sh
checks that the PR branch is up-to-date.
check_status.py
checks the status of EC2 instances.
setup.py
andci.test
configure the test cases to execute in the CI/CD workflow.
6.2. Auto RT¶
The Automated Regression Testing (AutoRT) system is a python program that automates the process of regression testing on NOAA HPC platforms. It contains the files in Table 6.1 below:
File Name |
Description |
---|---|
start_rt_auto.sh |
Verifies HPC name, sets the python paths |
rt_auto.py |
Python interface between the HPC and the github API |
jobs/bl.py |
Functions for the baseline job |
jobs/rt.py |
Functions for the regression test job |
6.2.1. AutoRT Workflow¶
On supported HPC systems, a cron job runs the start_rt_auto.sh
bash script every 15 minutes.
This script checks the HPC name and sets certain python paths. Then, it runs rt_auto.py
,
which uses the Github API (through pyGitHub) to check the labels on pull requests to
ufs-weather-model
. If a PR label matches the HPC name
(e.g., hera-intel-RT or cheyenne-gnu-BL), the label provides the HPC
with the compiler and job information to run a test or task on the machine.
If no PR label matches HPC name, the script exits.
For example, a PR labeled gaea-intel-BL
will be recognized by the HPC machine ‘Gaea’.
It will set the RT_COMPILER
variable to ‘intel’ and run the baseline creation script (bl.py
).
This script creats a job class that contains all information from the machine that the job will need to run.
That information is sent into the jobs/rt[bl].py
script.
rt.py
sets directories for storage, gets repo information, runs the regression test, and
completes any required post processing.
def run(job_obj):
logger = logging.getLogger('RT/RUN')
workdir = set_directories(job_obj)
branch, pr_repo_loc, repo_dir_str = clone_pr_repo(job_obj, workdir)
run_regression_test(job_obj, pr_repo_loc)
post_process(job_obj, pr_repo_loc, repo_dir_str, branch)
bl.py
: (similar to rt.py
) Adds functionality to create baselines before running regression testing.
def run(job_obj):
logger = logging.getLogger('BL/RUN')
workdir, rtbldir, blstore = set_directories(job_obj)
pr_repo_loc, repo_dir_str = clone_pr_repo(job_obj, workdir)
bldate = get_bl_date(job_obj, pr_repo_loc)
bldir = f'{blstore}/develop-{bldate}/{job_obj.compiler.upper()}'
bldirbool = check_for_bl_dir(bldir, job_obj)
run_regression_test(job_obj, pr_repo_loc)
post_process(job_obj, pr_repo_loc, repo_dir_str, rtbldir, bldir)