Scientific Computing

For system Python or other cases where “site-packages” is a non-writable directory, the pip --user option is necessary to install a Python package under the user home directory. However, if using Python virtualenv (with or without conda) the pip --user option is invalid. Environment variables set by Python indicate when a virtualenv is being used by Python currently.

“pip” is important for locally installed packages, since pip via pyproject.toml will automatically use the latest setuptools. This is quite important as too many user systems have too-old setuptools. The project’s pyproject.toml file should contain at least:

pyproject.toml:

[build-system]
requires = ["setuptools>=61.0.0", "wheel"]
build-backend = "setuptools.build_meta"

Detect Anaconda environment by existence of environment variable CONDA_PREFIX

CMakeLists.txt

find_package(Python COMPONENTS Interpreter REQUIRED)

# detect virtualenv and set Pip args accordingly
if(DEFINED ENV{VIRTUAL_ENV} OR DEFINED ENV{CONDA_PREFIX})
  set(_pip_args)
else()
  set(_pip_args "--user")
endif()

To install a package (named in CMake variable _pypkg) from PyPI:

# install PyPI Python package using pip
execute_process(COMMAND ${Python_EXECUTABLE} -m pip install ${_pip_args} ${_pypkg})

To install a local package in development mode (live changes):

execute_process(COMMAND ${Python_EXECUTABLE} -m pip install ${_pip_args} -e ${CMAKE_CURRENT_LIST_DIR})

CMake internally in Modules/FindPython/Support.cmake detects Python virtualenv.

One of the powerful uses of YaML with CI systems such as GitHub Actions (GHA) is the ability to setup job matrices. Job matrices allow deduplication of jobs with terse specification, encouraging better test platform and parameter coverage. GHA strategy matrix allows excluding jobs, including via exclusion arrays. An example strategy matrix below excludes shared build on macOS, due to bugs in third party library on macOS.

jobs:

  unix:

    strategy:
      matrix:
        shared: [true, false]
        img: [
          {os: ubuntu-latest, fc: gfortran},
          {os: macos-latest, fc: gfortran-13}
        ]
        exclude:
          - shared: true
            img: {os: macos-latest}

    runs-on: ${{ matrix.img.os }}
    env:
      FC: ${{ matrix.img.fc }}

This strategy results in 3 CI jobs:

• os=ubuntu-latest, shared=true, fc=gfortran
• os=macos-latest, shared=false, fc=gfortran-13
• os=ubuntu-latest, shared=false, fc=gfortran

Getting Python can be tricky for license-restricted users e.g. government and corporations. Building Python can be an arcane process without the automation of a high-level build system CMake. Python uses Autotools on most platforms except Windows where Visual Studio is used. The libraries need to be built with specific options and the forums are full of suggestions for tweaking Python build scripts etc.

The CMake project to build Python elides those issues for Linux/macOS platforms at least. It builds basic requirements of Python including expat, ffi, bzip2, xz, readline, zlib and more.

In CMake, undefined variables can evaluate as false in simple if(x) statements. For comparison operations like LESS GREATER, undefined variables do not operate like “false”.

if(x LESS 1)
  message(FATAL_ERROR "undefined not less 1")
endif()

No matter what value is compared to undefined variable “x”, the if() statement will not be true. As in most programming languages, a key best practice for CMake is to ensure variables are defined with a proper default value.

Building an Autotools project as a CMake ExternalProject saves the time of converting that other project to CMake. This technique makes it easy to automatically build that other project when it’s not easily installable otherwise, or you wish to build it optimized. This technique does not needlessly rebuild the ExternalProject each time the main CMake project is rebuilt.

It is useful to first check that Autotools is available to avoid emitting errors at build time. The goal is to emit errors about missing packages during build system configuration instead of during the build. We do this by including the file FindAutotools.cmake.

INSTALL_COMMAND

Note that “-j” option is NOT used to avoid race conditions in install scripts that might intermittently fail

CONFIGURE_HANDLED_BY_BUILD true

avoid constant reconfigure / rebuild.

“my_LIBRARY” is the known library file(s) built by the Autotools project.

include(GNUInstallDirs)
include(ExternalProject)

set_property(DIRECTORY PROPERTY EP_UPDATE_DISCONNECTED true)
# don't recheck for updates to the Git repo at subsequent CMake reconfigure

set(config_flags)  # parameters desired for ./configure of Autotools

set(my_LIBRARY ${CMAKE_INSTALL_FULL_LIBDIR}/${CMAKE_STATIC_LIBRARY_PREFIX}mylib${CMAKE_STATIC_LIBRARY_SUFFIX})

find_program(MAKE_EXECUTABLE NAMES gmake make mingw32-make REQUIRED)

ExternalProject_Add(mylib
URL https://github.invalid/username/archive.tar.bz2
CONFIGURE_HANDLED_BY_BUILD true
CONFIGURE_COMMAND <SOURCE_DIR>/configure ${config_flags}
BUILD_COMMAND ${MAKE_EXECUTABLE} -j
INSTALL_COMMAND ${MAKE_EXECUTABLE} install
TEST_COMMAND ""
BUILD_BYPRODUCTS ${my_LIBRARY}
)

add_library(mylib::mylib INTERFACE IMPORTED GLOBAL)
target_include_directories(mylib::mylib INTERFACE ${CMAKE_INSTALL_FULL_INCLUDEDIR})
target_link_libraries(mylib::mylib INTERFACE "${my_LIBRARY}")
# need the quotes to expand list
add_dependencies(mylib::mylib mylib)

For Ninja BUILD_BYPRODUCTS is necessary to avoid “ninja: error: “lib” needed by “target”, missing and no known rule to make it”

Some Autotools projects may need a “bootstrap” before “configure”. Add this script if needed:

ExternalProject_Add_Step(mylib
bootstrap
COMMAND <SOURCE_DIR>/bootstrap
DEPENDEES download
DEPENDERS configure
)

CMake ExternalProject allows building a wide variety of subprojects isolated from the main CMake project. For GNU Make Makefile projects, it is necessary to invoke the make command. However, there are several programs named “make” across operating systems. To help ensure the correct GNU Make is selected, we do:

find_program(MAKE_EXECUTABLE
NAMES gmake mingw32-make make
NAMES_PER_DIR
DOC "GNU Make")

A real-life example of CMake with Makefile ExternalProject has multiple Make invocations to build separate Make target groups, where later Make targets depend on the other Make targets being built first. We just show a snippet here for clarity, omitting definition of some of the obvious variables used.

CONFIGURE_COMMAND ""

since Make doesn’t have a configure step, so we must define this blank, as otherwise CMake will try to find a CMakeLists.txt in the external project code.

BUILD_COMMAND

builds the first target(s) that are required by targets in subsequent steps. If there’s no subsequent targets, this is the only build step.

INSTALL_COMMAND

Note that “-j” option is NOT used to avoid race conditions in install scripts that might intermittently fail.

BUILD_BYPRODUCTS

In general we point this at the “installed” files, as otherwise “ninja: error: “lib” needed by “target”, missing and no known rule to make it”. Ninja is stricter than Make about the target to source graph.

include(GNUInstallDirs)

set_property(DIRECTORY PROPERTY EP_UPDATE_DISCONNECTED true)
# don't recheck for updates to the Git repo at subsequent CMake reconfigure

find_program(MAKE_EXECUTABLE NAMES gmake make mingw32-make REQUIRED)

set(my_LIBRARY ${CMAKE_INSTALL_FULL_LIBDIR}/${CMAKE_STATIC_LIBRARY_PREFIX}mylib${CMAKE_STATIC_LIBRARY_SUFFIX})

ExternalProject_Add(mylib
URL https://github.invalid/username/archive.tar.bz2
CONFIGURE_COMMAND ""
BUILD_COMMAND ${MAKE_EXECUTABLE} -j -C <SOURCE_DIR>
INSTALL_COMMAND ${MAKE_EXECUTABLE} -C <SOURCE_DIR> install prefix=${CMAKE_INSTALL_PREFIX}
BUILD_BYPRODUCTS ${my_LIBRARY}
)

CI services typically have a per-job timeout parameter that saves compute resources from being wasted. Overall CI system performance is improved by abandoning jobs taking an unexpectedly long time to run. A typical CI experience is that during busy times, certain tasks like downloading can take much 10x or more longer than usual. A balance is struck between expected task time and timeout limit.

repository install

The apt install or brew install operations that access a remote repository are often run in CI tasks. These tasks might take 10-30 seconds normally, but during heavy CI use times they may take about 10 minutes. This makes setting timeouts less intuitive with jobs that only take a few minutes or less build / test time. Empirically, we find with GitHub Actions that making the overall job timeout 15 minutes or so allows for the sometimes 10 minutes peak download time tolerable.

jobs:

  linux:
    runs-on: ubuntu-latest
    timeout-minutes: 15

    steps:
    - uses: actions/checkout

    - name: install prereqs
      run: |
        sudo apt update
        sudo apt install libopenmpi-dev

    - run: ...

Python pip

We often set a “pip” timeout of 1 or 2 minutes to ensure the CI is using wheels instead of compiling from source, which can take tens of minutes or fail. A job-level timeout can likewise be set.

In this case, rather than setting a job-level timeout-minutes: 6, we know pip for this project takes much less than 1 minute, so we only have to wait one minute if pip goes awry. This becomes important for projects with lots of CI jobs and workflows, you don’t want to have to manually wade through hundreds or thousands of error messages.

jobs:

  linux:
    runs-on: ubuntu-latest
    timeout-minutes: 10

    steps:
    - uses: actions/checkout
    - uses: actions/setup-python
      with:
        python-version: '3.x'

    - run: python -m pip install .[tests]
      timeout-minutes: 1

    - run: pytest
      timeout-minutes: 5

Windows-based devices using RS232 or USB to RS232 connection may work on Linux (including Windows Subsystem for Linux) using WINE. The examples assume a USB to RS232 adapter.

Access serial ports without needing sudo:

adduser $(whoami) dialout

Logout and login again. Plug in USB-RS232 adapter.

ls /dev/ttyUSB*

The USB to serial converter will probably show up as /dev/ttyUSB0 Find the corresponding WINE device port by

ls -l ~/.wine/dosdevices

If necessary, change the serial port mapping in WINE via regedit.

Physical serial ports on Linux such computers with 9-pin / 25-pin RS232/RS485 ports are accessed by:

apt install setserial

setserial -g /dev/ttyS* | grep -v unknown

Related: setup serial port DOSBox on Linux

CMake requires target parameter WINDOWS_EXPORT_ALL_SYMBOLS set to true to properly use shared libraries on Windows with Visual Studio or Intel oneAPI (which uses Visual Studio as a backend on Windows). For convenience CMAKE_WINDOWS_EXPORT_ALL_SYMBOLS sets all targets default parameter. Without this parameter, linking fails because the “.lib” import library isn’t created with the exported shared library symbols.

An example CMakeLists.txt where this parameter is needed to build successfully with Visual Studio or Intel oneAPI on Windows:

set(BUILD_SHARED_LIBS true)

set(CMAKE_WINDOWS_EXPORT_ALL_SYMBOLS true)
# must be before all targets

add_library(a a.c)
add_library(b b.c)

target_link_libraries(b PRIVATE a)

It’s also possible to set this parameter for individual targets like:

set_property(TARGET a PROPERTY WINDOWS_EXPORT_ALL_SYMBOLS true)

CMake normal variables can be used without ${} evaluation in several contexts including if() statements. Cache and Environment variables do need ${} enclosure even in if() statements to be evaluated.

Example: environment variable “CI” to detect if CMake is running on a CI or not.

# ENV{...} is always false.
# $ENV{...} is correct syntax

if($ENV{CI})
  message(STATUS "CI value: $ENV{CI}")
endif()

When using if() to check for variable existence, then ${} is not used.

if(DEFINED ENV{CI})
  message(STATUS "CI value: $ENV{CI}")
endif()