CMake RESOURCE_LOCK vs. RUN_SERIAL advantages

CMake (via CTest) can run tests in parallel via the option

ctest --parallel 4

where “4” is the number of tests to run in parallel–set it approximately equal to the number of CPU cores in your system typically. Some tests need to be run not in parallel, for example tests using MPI that use lots of CPU cores, or tests that use a lot of RAM, or tests that must access a common file or hardware device. We have found that using the RUN_SERIAL makes whole groups of tests run sequentially instead of individually running sequentially when fixtures are used. That is, all the FIXTURES_SETUP run, then all FIXTURES_REQUIRED that have RUN_SERIAL. This is not necessarily desired, because we had consuming fixtures that didn’t have to wait for all the fixtures to be setup.

We found that using RESOURCE_LOCK did not suffer from this issue, and allows the proper test dependencies and the expected parallelism.

Example

For simplicity we omit the necessary add_test() and just show the properties.

The test has an MPI-using quick setup “Quick1” and then a long test “Long1” also using MPI. Finally, we have a quick Python script “Script1” checking the output.

In the real setup, we have Quick1, Quick2, … QuickN and so on. When we used RUN_SERIAL, we had to wait for ALL Quick* before Long* would start. With RESOURCE_LOCK the tests intermingle, making better use of CPU particularly on large CPU count systems, and with lots of tests.

The name “cpu_mpi” is arbitrary like the other names.

set_tests_properties(Quick1 PROPERTIES
  RESOURCE_LOCK cpu_mpi
  FIXTURES_SETUP Q1)

set_tests_properties(Long1 PROPERTIES
  RESOURCE_LOCK cpu_mpi
  FIXTURES_REQUIRED Q1
  FIXTURES_SETUP L1)

set_tests_properties(Script1 PROPERTIES
  FIXTURES_REQUIRED L1)

Notes

CMake Resource Groups are orthogonal to Resource Locks, and are much more complicated to use. There may be some systems that would benefit from Groups, but many can just use the simple Locks.