C++ Matrix
The C Matrix Action is designed to automate the generation of a comprehensive test matrix for your C libraries given their requirements. It simplifies the process of defining the contract of what platforms your library supports and testing your project across a proper selected subset of revelant compiler versions and platforms.
With this action, you can define a set of requirements to test your C++ library. It will then generate a test matrix by combining the specified requirements into a fractional factorial design to ensure all proper combinations are tested in a systematic manner. This approach significantly increases the chances of catching compatibility issues early on and helps in delivering a robust and reliable library to users.
Usage
The action should be run as extra initial “setup” job in your workflow. The job will run the action and
output matrix, which is a JSON string containing the whole matrix. This matrix should be output of your
first setup job.
In your second build job, you can attribute the entire matrix to the strategy.matrix.include parameter
and create your workflow as usual with the parameters from the matrix:
jobs:
cpp-matrix:
runs-on: ubuntu-latest
name: Generate Test Matrix
outputs:
matrix: ${{ steps.cpp-matrix.outputs.matrix }}
steps:
- name: Generate Test Matrix
uses: alandefreitas/cpp-actions/cpp-matrix@v1.8.2
id: cpp-matrix
with:
standards: '>=11'
build:
needs: cpp-matrix
strategy:
fail-fast: false
matrix:
include: fromJSON(needs.cpp-matrix.outputs.matrix)
# use matrix entries
name: ${{ matrix.name }}
runs-on: ${{ matrix.runs-on }}
container: ${{ matrix.container }}
steps:
- name: Clone library
uses: actions/checkout@v3
- name: Setup C++ Compiler
uses: alandefreitas/cpp-actions/setup-cpp@v1.8.2
id: setup-cpp
with:
compiler: ${{ matrix.compiler }}
version: ${{ matrix.version }}
- name: CMake Workflow
uses: ./cmake-workflow
with:
cc: ${{ steps.setup-cpp.outputs.cc || matrix.cc }}
cxx: ${{ steps.setup-cpp.outputs.cxx || matrix.cxx }}
cxxstd: ${{ matrix.cxxstd }}
cxxflags: ${{ matrix.cxxstd }}
# And you've safely tested your C++ library just like that...Factors
The input parameters allow factors to be defined for each compiler. These factors can be defined as latest factors (which are duplicated and applied to the latest version of each compiler) or as regular factors (which are injected into intermediary versions of each compiler).
See the Factors Section for a better understanding of the motivation for these factor types.
Matrix entries
The values in each matrix entry include a few categories of keys: the main keys, auxiliary keys, factor keys, and suggestions.
The main keys include information such as the compiler name and version. Auxiliary keys include information that is useful for filtering and sorting the matrix, such as information about the relative position of an entry in the matrix. Factor keys include information about the factors that might have been applied to the entry.
Suggestions include opinionated recommendations for other parameters of your workflow as a
starting point for your test matrix. Some examples of keys including suggestions are
runs-on, container, generator`, b2-toolset, build-type, ccflags, cxxflags,
env, and install.
These suggestions often need to be customized and this could not be different, since there is a myriad of ways in which libraries should be tested. Common ways to customize these are:
-
Ignoring suggestions that are not applicable to your use case
-
Using the corresponding action input to customize the value of these fields (see action input parameters)
-
Use GitHub Actions Expressions to generate new values from existing values
-
Use a custom bash step to generate new values from existing values
-
Create a custom script to read the complete matrix and generate a new matrix with the desired values
For instance, if the recommendation of running a workflow in the container ubuntu:22.04 is not appropriate
because you’re going to use a container that already has your dependencies, you could set the runs-on input
parameter to:
- name: Generate Test Matrix
# ...
with:
# ...
containers: |
gcc: my-ubuntu:22.04-containerThis would replace the recommendation for ubuntu:22.04 with my-ubuntu:22.04-container for all entries
with the compiler gcc.
In some cases, you could use a expression directly in the workflow to use a different value for the container:
# In your main workflow
containers: ${{ matrix.runs-on == 'ubuntu-22.04' && 'my-ubuntu:22.04-container' || matrix.container }}This would replace the recommendation for ubuntu:22.04 with my-ubuntu:22.04-container for all entries.
If you’re familiar with GitHub Actions and bash, you can also use a custom bash step to generate new values from existing values or write a complete script to customize the matrix.
If none of these options is enough for the library requirements, the action also prints the complete matrix in YAML format, so it can be copy/pasted into the workflow as a starting point to be customized.
Ordering entries
Entries are ordered in the matrix according to the following criteria:
1) Latest versions of each compiler 2) Compilers with single versions 3) Oldest versions of each compiler 4) Entries with factors 5) Intermediary versions of each compiler
The input parameter generate-summary can be used to generate a summary of the matrix with the
entries ordered according to these criteria.
Semver ranges
The requirement parameters support semver ranges, including range disjunctions. For instance, consider the default
compiler range clang >=3.8, which might test your library with too many versions of clang. You could remove
intermediary versions of clang with the range clang >=3.8 <5 || >=10, which ensure old versions are
recent versions are covered, while eliminating intermediary versions.
Semver ranges can also help when a bug is found in a specific compiler version. Consider someone has reported
a bug in GCC 8.1, and you want to keep track of that. The semver range gcc >=4.8 <7 || 8.1 || >=10 would ensure
version GCC8.1 specifically, and not any other version is the range >=8.0.0 <9.0.0 is tested.
Dynamic matrices
In some cases, it might be useful to test different matrices based on the conditions of the workflow. These conditions might be the event_type and the types of files changed in the commit triggering the workflow. The workflows might vary between disabling tests for changes that don’t affect these certain files, running a reduced subset of tests for less important changes, or enabling extra tests, such as documentation tests in case only these files have been changed. This might be useful in terms of performance, costs, and safety.
While this is hard to achieve with hard-coded matrices, this action makes variable matrices very easy to achieve.
Simply define a previous step to determine the matrix parameters and use it to generate the matrix.
For instance, the compilers parameters could be replaced from gcc >=4.8 to
gcc >=4.8 <6.0 ${{ github.event_name == 'push' && '|| >=6.0 <10' }} || >=11 to only test intermediary
GCC versions >=6.0 <10 when in push events.
A more powerful strategy to save resources is analyzing files changed in the workflow and considering these files to determine what the matrix should be according to project conventions.
Next steps
After setting creating the test matrix, the next step in your workflow should usually be Setup C++.
Example
steps:
- name: Generate Test Matrix
uses: alandefreitas/cpp-actions/cpp-matrix@v1.8.2
id: cpp-matrix
with:
standards: '>=11'
compilers: 'gcc >=4.8 <6 || >=9
clang >=3.8 <6 || >11
msvc >=14
apple-clang *
mingw *
clang-cl *
'
latest-factors: 'msvc ASan
gcc Coverage TSan UBSan Fetch-Content
clang Fetch-Content
'
factors: 'gcc ASan Shared No-Deps
msvc Shared x86
clang Time-Trace ASan+UBSan
mingw Shared
'Input Parameters
Parameter |
Description |
Default |
|
A list of compilers to be tested. Each compiler can be complemented with its semver version requirements to be tested. When the compiler version requirements are provided, the action will break the requirements into subsets of major versions to be tested. When no version is provided, the '*' semver requirement is assumed. The action can identifies subsets of compiler versions for GCC, Clang, and MSVC. For any other compilers, the version requirements will passthrough to the output. |
|
|
A semver range describing what C standards should be tested. The compiler ranges are adjusted to only include compilers that support any subrange of these requirements. These requirements can include C standards as 2 or 4 digits versions, such as 11, 2011, 98, or 1998. 2 digit versions are normalized into the 4 digits form so that 11 > 98 (2011 > 1998). |
|
|
The maximum number of standards to be tested with each compiler. For instance, if 'max-standards' is 2 and the compiler supports '11,14,17,20,23' given the in the standard requirements, the standards 20,23 will be tested by this compiler. |
|
|
The factors to be tested with the latest versions of each compiler. For each factor in this list, the entry with the latest version of a compiler will be duplicated with an entry that sets this factor to true. Other entries will also include this factor as false. The following factors are considered special: 'asan', 'ubsan', 'msan', 'tsan', and 'coverage'. When these factors are defined in an entry, its 'ccflags', 'cxxflags', and 'linkflags' value are also modified to include the suggested flags for factor. |
|
|
The factors to be tested with other versions of each compiler. Each factor in this list will be injected into a version of the compiler that is not the latest version. An entry with the latest version of the compiler will be duplicated with this factor if there are no entries left to inject the factor. Other entries will also include this factor as false. |
|
|
The factors to be tested with all combinations of other factors. When combinatorial factors are defined,
for each entry in the matrix, a new entry will be created with the factors in this list set to For instance, if the library can be built both in "Standalone" mode and with dependencies, the factor 'Standalone'
can be added to this list to duplicate all entries. Each copy would include a "Standalone" factor set to
Typically, it is advisable to steer clear of combinatorial factors to prevent a combinatorial explosion. It’s usually better to only test the combinations that are expected to be used in practice and include an extra steps in the workflow to test any combinatorial factors. For instance, if the library can be built both in "Standalone" mode and with dependencies, its workflow can simply include an extra step to also test the library in "Standalone" mode and keep the step to test the library with dependencies. This is usually safer and cheaper than duplicating the entire matrix to test all combinations of these factors and also allows steps to be skipped when the library is not expected to be built in a given mode. For instance, testing a library on Standalone mode might not be necessary when the library is being tested with intermediary compilers. |
|
|
A multi-line list of github runner images to be used with each range of compiler version. Each line has the format:
For instance, Omitting When the runner image is specified, a container is only be suggested for the entries if the When the runner image is unspecified, the action will infer the runner image and potentially a container from the compiler name and its version. |
|
|
A multi-line list of docker containers to be used with each range of compiler version. Each line has the format:
For instance, Omitting When the container is specified for that compiler version and the When the container is unspecified, the action can still infer a container for the compiler version according
to the rules defined in the |
|
|
A multi-line list of cmake generators to be used with each range of compiler version. Each line has the format:
For instance, Omitting When the generator is unspecified, the action will infer the generator from the compiler name and its version. |
|
|
A multi-line list of cmake generator toolsets to be used with each range of compiler version. Each line has the format:
For instance, Omitting When the generator toolset is unspecified, the action will infer the generator toolset from the compiler name and its version. |
|
|
A multi-line list of b2 toolsets to be used with each range of compiler version. Each line has the format:
For instance, Omitting When the toolset is unspecified, the action will infer the toolset from the compiler name and its version. |
|
|
A multi-line list of C compiler flags to be used with each range of compiler version. Each line has the format:
For instance, Omitting When the flag is unspecified, the action will infer the flag from the compiler name and its version. |
|
|
A multi-line list of C compiler flags to be used with each range of compiler version. Each line has the format: `<compiler-name>[ <compiler-range|compiler-factor>]: <cxxflags>` For instance, `gcc >=13.1: -O3` indicates that the C compiler flag Omitting When the flag is unspecified, the action will infer the flag from the compiler name and its version. |
|
|
A multi-line list of packages to be installed with each range of compiler version. Each line has the format:
For instance, Omitting When the package is unspecified, the action will infer the package from the compiler name and its version. |
|
|
A multi-line list of triplets to be used with each range of compiler version. Each line has the format:
For instance, Omitting When the triplet is unspecified, the action will infer the triplet from the compiler name and its version. |
|
|
A multi-line list of build types to be used with each range of compiler version. Each line has the format:
For instance, Omitting When the build type is unspecified, the action will infer the build type from the compiler name and its version. |
|
|
Determine the default build type to suggest when testing with sanitizers. |
|
|
Determine the default build type to suggest when testing with x86. |
|
|
Determine whether to use containers whenever possible to run the tests. By using containers for all jobs, the workflow can be more stable and reproducible. For instance, without containers an existing workflow cannot start to fail because of a change in the GitHub runner environments. However, this comes at a cost of initial setup time. Some existing workflows can also break when moving to containers because existing assumptions about tools available in the runner environment are no longer valid. When the value is false, the action will still use containers when needed. This may happen because the compiler is not available in the runner image or when there’s a reported conflict between compilers in the runner image. |
|
|
Log the generated matrix as a JSON string. The is useful for debugging purposes and when transitioning to a workflow that uses a hard-coded matrix. |
|
|
Generate summary with the complete matrix. |
|
|
Trace commands executed by the action. |
|
Outputs
Output |
Description |
|
The test matrix is an array of dictionaries, where each entry represents a combination of compiler version and factors to be tested. Each entry in the test matrix dictionary contains key-value pairs in the following categories: Basic fields: - - - - - Auxiliary: - - - - - - - - Factors: - Suggestions: - - - - - - - - - - - |