C++ Matrix
Motivation
Testing C++ libraries can be a challenging and time-consuming task, especially when considering the various compiler versions, error behaviors, and platform dependencies. A library that works flawlessly with one compiler will almost certainly fail to build with a different compiler version, not mention different compilers and platforms.
This has a viral effect with enormous implications to the C++ ecosystem. If your library doesn’t properly test and support all platforms in its manifest, this support is also broken for any projects depending on your library. With some notable exceptions, this is currently the case with most small standalone libraries one can find on GitHub. These are libraries that could be very useful otherwise, so avoid dependencies altogether is not a reasonable to the problem, since we will never get anywhere meaningful if we’re not able to stand to the shoulder of giants.
C++ Test Matrix
The C Matrix Action is designed to automate the generation of a comprehensive test matrix for your C libraries given their requirements. It simplifies the process of defining the contract of what platforms your library supports and testing your project across a proper selected subset of revelant compiler versions and platforms.
With this action, you can define a set of requirements to test your C++ library. It will then generate a test matrix by combining the specified requirements into a fractional factorial design to ensure all proper combinations are tested in a systematic manner. This approach significantly increases the chances of catching compatibility issues early on and helps in delivering a robust and reliable library to users.
Usage
The action should be run as extra initial “setup” job in your workflow. The job will run the action and
output matrix, which is a JSON string containing the whole matrix. This matrix should be output of your
first setup job.
In your second build job, you can attribute the entire matrix to the strategy.matrix.include parameter
and create your workflow as usual with the parameters from the matrix:
jobs:
cpp-matrix:
runs-on: ubuntu-latest
name: Generate Test Matrix
outputs:
matrix: ${{ steps.cpp-matrix.outputs.matrix }}
steps:
- name: Generate Test Matrix
uses: alandefreitas/cpp-actions/cpp-matrix@v1.6.1
id: cpp-matrix
with:
standards: '>=11'
build:
needs: cpp-matrix
strategy:
fail-fast: false
matrix:
include: fromJSON(needs.cpp-matrix.outputs.matrix)
# use matrix entries
name: ${{ matrix.name }}
runs-on: ${{ matrix.runs-on }}
container: ${{ matrix.container }}
steps:
- name: Clone library
uses: actions/checkout@v3
- name: Setup C++ Compiler
uses: alandefreitas/cpp-actions/setup-cpp@v1.6.1
id: setup-cpp
with:
compiler: ${{ matrix.compiler }}
version: ${{ matrix.version }}
- name: CMake Workflow
uses: ./cmake-workflow
with:
cc: ${{ steps.setup-cpp.outputs.cc || matrix.cc }}
cxx: ${{ steps.setup-cpp.outputs.cxx || matrix.cxx }}
cxxstd: ${{ matrix.cxxstd }}
cxxflags: ${{ matrix.cxxstd }}
# And you've safely tested your C++ library just like that...Variable matrices
In some cases, it might be useful to test different matrices based on the conditions of the workflow. These conditions might be the event_type and the types of files changed in the commit triggering the workflow. The workflows might vary between disabling tests for changes that don’t affect these certain files, running a reduced subset of tests for less important changes, or enabling extra tests, such as documentation tests in case only these files have been changed. This might be useful in terms of performance, costs, and safety.
While this is hard to achieve with hard-coded matrices, this action makes variable matrices very easy to achieve.
Simply define a previous step to determine the matrix parameters and use it to generate the matrix.
For instance, the compilers parameters could be replaced from gcc >=4.8 to
gcc >=4.8 <6.0 ${{ github.event_name == 'push' && '|| >=6.0 <10' }} || >=11 to only test intermediary
GCC versions >=6.0 <10 when in push events.
A more powerful strategy to save resources is analyzing files changed in the workflow and considering these files to determine what the matrix should be according to project conventions.
About matrix recommendations
The values in each matrix entry very opinionated recommendations as a starting point for your test matrix. This could not be different, since there is a myriad of ways in which build parameters can be tested.
Ignoring recommendations
If a given recommendation does not apply to your use case, the first option is simply to ignore that
entry key. For instance, matrix.install includes suggested packages that should be installed, but these
can be ignored if a custom container with these dependencies is already provided.
Some factors always need to be adjusted when basic assumptions about an initial library are not valid.
For instance, the matrix will include the factor matrix.shared by default. If the only use case for your
library is as a shared library, then this Shared factor can be removed from the matrix and you can configure
all workflows to build a shared library.
Adjusting semver ranges
The input parameters support semver ranges, including range disjunctions. For instance, consider the default
compiler range clang >=3.8, which might test your library with too many versions of clang. You could remove
intermediary versions of clang with the range clang >=3.8 <5 || >=10, which ensure old versions are
recent versions are covered, while eliminating intermediary versions.
Also when a bug is found in a specific version, semver ranges can also help. Consider someone has reported
a bug in GCC 8.1, and you want to keep track of that. The semver range gcc >=4.8 <7 || 8.1 || >=10 would ensure
version GCC8.1 specifically, and not any other version is the range >=8.0.0 <9.0.0 is tested.
Overriding recommendations
When the general recommendation does not work, but there’s a mapping between the recommendation and the expected
values, you can use github action expressions to directly replace these values. For instance, if the recommendation
of running a workflow in the runner image ubuntu-22.04 is not appropriate because you’re going to use a container
that already has your dependencies, you could set the job runs-on property to ubuntu-22.04 instead of
matrix.runs-on and set the container property to
( matrix.runs-on == 'ubuntu-22.04' && 'my-ubuntu-22.04-container' ) || matrix.container ), which replaces all
occurrences of ubuntu-22.04 with my-ubuntu-22.04-container.
Auxiliary keys
Overriding recommendations can be verbose when there are too many conditions involved. The matrix provides
auxiliary keys to make this easier. For instance, consider a matrix entry with the factor matrix.cxxstd equal
to 17,20. If you want to have two versions of tests, where one of them does not go through all C++ standards,
you can use the matrix.latest-cxxstd, which will give you 20 without having to split matrix.cxxstd and take
its last value.
Other auxiliary keys have broader meaning, so they can be used in more general situations. For instance,
it’s common to build release artifacts for a subset of entries in the matrix. This subset usually represents
the latest version of each compiler without any factors applied. In that case, the factor matrix.is-main can
be use to determine if the release artifacts should be generated. The factor matrix.is-main determines if
the current entry is the latest version of a compiler but not one of the replicates of the latest version
with factors applied.
Use scripts
In more complex cases, it might be worth considering that this action only returns a json representation of the test matrix. If deeper modifications in the matrix are required, a following step including a script to adjust the matrix is always possible. In this scenario, the matrix will still fetch updated information about compiler versions and requirements while the script will only adjust local requirements. In most cases, this is unnecessary as the approaches above tend to have good results.
Next steps
After setting creating the test matrix, the next step in your workflow should usually be Setup C++.
Example
steps:
- name: Generate Test Matrix
uses: alandefreitas/cpp-actions/cpp-matrix@v1.6.1
id: cpp-matrix
with:
standards: '>=11'
compilers: 'gcc >=4.8 <6 || >=9
clang >=3.8 <6 || >11
msvc >=14
apple-clang *
mingw *
clang-cl *
'
latest-factors: 'gcc Coverage TSan UBSan Fetch-Content
clang Fetch-Content
'
factors: 'gcc Asan Shared No-Deps
msvc Shared x86
clang Time-Trace
mingw Shared
'Input Parameters
Parameter |
Description |
Default |
|
A list of compilers to be tested. Each compiler can be complemented with its semver version requirements to be tested. When the compiler version requirements are provided, the action will break the requirements into subsets of major versions to be tested. When no version is provided, the '*' semver requirement is assumed. The action can identifies subsets of compiler versions for GCC, Clang, and MSVC. For any other compilers, the version requirements will passthrough to the output. |
|
|
A semver range describing what C standards should be tested. The compiler ranges are adjusted to only include compilers that support any subrange of these requirements. These requirements can include C standards as 2 or 4 digits versions, such as 11, 2011, 98, or 1998. 2 digit versions are normalized into the 4 digits form so that 11 > 98 (2011 > 1998). |
|
|
The maximum number of standards to be tested with each compiler. For instance, if 'max-standards' is 2 and the compiler supports '11,14,17,20,23' given the in the standard requirements, the standards 20,23 will be tested by this compiler. |
|
|
The factors to be tested with the latest versions of each compiler. For each factor in this list, the entry with the latest version of a compiler will be duplicated with an entry that sets this factor to true. Other entries will also include this factor as false. The following factors are considered special: 'asan', 'ubsan', 'msan', 'tsan', and 'coverage'. When these factors are defined in an entry, its 'ccflags', 'cxxflags', and 'linkflags' value are also modified to include the suggested flags for factor. |
|
|
The factors to be tested with other versions of each compiler. Each factor in this list will be injected into a version of the compiler that is not the latest version. An entry with the latest version of the compiler will be duplicated with this factor if there are no entries left to inject the factor. Other entries will also include this factor as false. |
|
|
Generate summary with the complete matrix. |
|
|
Trace commands executed by the action. |
|
Outputs
Output |
Description |
|
The test matrix is an array of dictionaries, where each entry represents a combination of compiler version and factors to be tested. Each entry in the test matrix dictionary contains the following key-value pairs: - - - - - - - - - - - - - - - - - - - - - |