Scientific Computing

Find executable path in Python

The full path to executables on the system Path (and cwd on Windows) are discovered by Python shutil.which. On Windows, environment variable PATHEXT is used to search filename suffixes if not specified at the input to shutil.which().

Shell aliases are not found by shutil.which() since the shell is not invoked. Instead append the directory of the desired executable to environment variable PATH, or specify it in shutil.which(..., path="/path/to/exe").

import shutil

# None if executable not found
exe = shutil.which('ls')

Since shutil.which() returns None for non-found executable it is convenient for pytest.mark.skipif

For programs not on PATH where the executable path is known:

shutil.which('myexe', path="/path/to/myexe")

Install Gfortran or Flang compiler on macOS Homebrew

Homebrew can install Fortran compilers including GCC and LLVM Flang.

Gfortran

Gfortran comes with GCC Homebrew package:

brew install gcc

As a complete C / C++ / Fortran compiler package, Gfortran doesn’t require additional flags or environment variables.

To use GCC compilers, source a script like:

p=$(brew --prefix gcc)/bin
v=14

export CC=$p/gcc-$v CXX=$p/g++-$v FC=$p/gfortran-$v

# to avoid GCC include errors -- MacOSX15.sdk incompatable at the moment with Homebrew GCC
export SDKROOT=/Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk/

where v=14 is the major version number of the GCC compiler installed. The SDKROOT line may be necessary when Homebrew GCC package hasn’t yet enabled the latest SDK–adjust to suite the system.

LLVM Flang

LLVM Flang is a separate package from the LLVM C/C++ compilers:

brew install flang

To use Flang compiler (works with Clang, AppleClang, GCC C-C++ compilers), source a script like:

export FC=$(brew --prefix flang)/bin/flang-new

To use LLVM Clang with Flang, source a script like:

p=$(brew --prefix llvm)/bin
export CC=$p/clang CXX=$p/clang++

export FC=$(brew --prefix flang)/bin/flang-new

Troubleshooting

When a new compiler version or macOS version or Xcode SDK is released, it may be necessary to adjust the environment variables or flags temporarily until Homebrew updates the package.

Some examples to try if needed:

  • path to libSystem.tbd and libc++.tbd

    export LIBRARY_PATH=/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/lib
  • path to the C++ standard library

    export LDFLAGS=-lc++

CMake find_program script

CMake find_program does not generally consider NAMES parameter to have file suffixes unless manually specified. For Windows, .com and .exe file suffixes are considered, with search order:

  1. .com
  2. .exe
  3. no suffix

If on Windows and an executable “hello.exe” or “hello.com” exists, then CMake will find it. CMake would NOT find “hello.exe” on non-Windows platforms, where no file suffix is expected.

The full path to executables on the system Path (and cwd on Windows) are found by find_program(). Shell aliases are not found since the shell is not invoked. Instead specify find_program(... HINTS /path/to/exe).

NOTE: CMAKE_EXECUTABLE_SUFFIX ONLY affects find_program() in CMake role PROJECT

find_program(v NAMES hello)

Shell scripts of any file suffix on any operating system are found iff:

  1. (non-Windows) script file executable flag is set (find_program general requirement)
  2. script file suffix is specified as part of find_program(… NAMES) parameter

A complete standalone example:

Iterate Matlab versions with CMake

These techniques work with any versioned program or library. Here we use Matlab as an example. CMake find_package with a version range would be used to simply select from a known-working version range.

Many Matlab codes require a modern version of Matlab. It’s possible to select from an arbitrary min…max range of Matlab versions with CMake FindMatlab as follows. This technique works with other versioned programs and libraries as well.

foreach(v IN ITEMS 23.2 24.1 24.2)
  find_package(Matlab ${v} EXACT COMPONENTS MAIN_PROGRAM)
  if(Matlab_FOUND)
    add_test(NAME matlab-${v}
      COMMAND ${Matlab_MAIN_PROGRAM} -batch "buildtool"
      WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}
      )
  endif()
endforeach()

Git SSL certificate location

Git typically works for HTTPS URLs using the default configuration. If certificates are not working, try configuring Git with the system SSL certificate location.

git config --global http.sslCAInfo /path/to/cert.crt

Git environment variable “GIT_SSL_CAINFO” can override this. This can be used to override missing or incorrect SSL certificate locations for Git operations in programs like Matlab.

To disable SSL certificate verification, set environment variable “GIT_SSL_NO_VERIFY” to “1”, but be aware of the security implications e.g. MITM.

Free C C++ Fortran compiler families

Several modern, currently-supported compiler families are free-to-use for C, C++ and Fortran.

GCC has broad support of modern standards on a very wide range of computing platforms. GCC’s downside in some cases can be slower runtime performance than compilers having less broad language and platform support.

Compiler C C++ Fortran
GNU gcc: C23 g++ C++26 gfortran: F2018
Intel oneAPI icx: C23 icpx: C++23 ifx: F2023
LLVM clang: C17 clang++: C++26 flang-f18: F2018
AOCC clang: C17 clang++: C++17 flang: F2008
NVIDIA HPC SDK nvc: C11 nvc++: C++23 nvfortran: F2003
IBM OpenXL xlc: C17 xlc++: C++26 xlf: F2018

Intel oneAPI compilers are free to use for any user. The Intel performance libraries like MKL, IPP, and TBB are available at no cost.


LLVM Clang and Flang have significant industry support, including from Nvidia, and are known for high performance.


AMD AOCC LLVM compiler is tuned for AMD CPUs.


NVIDIA HPC SDK is free to use. A key feature of the HPC SDK compilers is intrinsic support for CUDA Fortran.


IBM OpenXL LLVM-based compilers are currently for POWER CPUs only e.g. ppc64le. IBM OpenXL compilers do not work with a typical x86-based computer.

CMake macOS Xcode Environment

Related: GCC / Clang header clash on macOS


When macOS, Xcode, or Command Line Tools upgrades, build directories (including for CMake-based projects) often need to be refreshed. If the user has set custom environment variables concerning Xcode, they may need to be updated as well. Here are some important environment variables and CMake variables to check if problems occur after upgrading.

For a simple CMake project on macOS, CMakeCache.txt might include:

CMAKE_OSX_DEPLOYMENT_TARGET:STRING=

CMAKE_OSX_SYSROOT:PATH=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX*.sdk

Having multiple directories under the following is fine:

  • /Library/Developer/CommandLineTools/SDKs
  • /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs

CMake can select the appropriate SDK by the user specifying environment variable SDKROOT or by selecting the SDK:

xcode-select --switch

The XCode versions that Homebrew targets can be found in xcode.rb.

Workarounds

If Homebrew GCC breaks after upgrading Xcode or Command Line Tools, try specifying an older SDK. For example, if g++-14 main.cpp -v shows a different (older) SDK than CMake and it works, try specifying that SDK in environment variable SDKROOT. Note that the SDK version corresponds to macOS version, not the XCode version. For example, if the latest SDK is MacOSX14.4.sdk, try using MacOSX13.3.sdk in “~/gcc.sh”:

export CC=gcc-14 CXX=g++-14 FC=gfortran-14

export SDKROOT=/Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk/

and then source ~/gcc.sh before running cmake with a fresh build directory.

If a CMake build step fails, try copy-pasting the command and removing the -isysroot portion of the command. This is a clear clue the older SDK is (at least temporarily) needed till Homebrew updates its GCC formula.

GCC will tell where included files are coming from by adding the gcc -H flag. This tells what to specify for environment variable SDKROOT.

Example minimal project

Zoom audio toggle indicator

As per the Release Notes:

The Zoom app offers an option to play audio tones when muting or unmuting the microphone, providing enhanced accessibility for users with visual impairments or other disabilities. This setting is OFF by default and can be enabled from the Zoom Workplace app under Settings then Audio. When enabled, users will hear a tone, audible only to them, indicating the mute or unmute action.

CMake variables in SCRIPT role

When running CMake standalone scripts like cmake -P script.cmake this is the SCRIPT CMake role. Not all CMake information variables are set in SCRIPT role, in particular, the CMAKE_HOST* and CMAKE_SYSTEM* variables are not set as they are in PROJECT role.

This is a workaround for cmake -P SCRIPT role to get the CMAKE_HOST_* variables. It uses undocumented CMake-internal scripts, but they’ve been present since 2012 and may be unlikely to change.

message(STATUS "CMake ${CMAKE_VERSION}")

get_property(cmake_role GLOBAL PROPERTY CMAKE_ROLE)
if(cmake_role STREQUAL "SCRIPT")
  set(CMAKE_PLATFORM_INFO_DIR ${CMAKE_BINARY_DIR}${CMAKE_FILES_DIRECTORY})
  # define CMAKE_HOST*, CMAKE_SYSTEM*, etc.
  include(${CMAKE_ROOT}/Modules/CMakeDetermineSystem.cmake)
  # set booleans like CYGWIN
  include(${CMAKE_ROOT}/Modules/CMakeSystemSpecificInitialize.cmake)

  # needed by Modules/Platform/*.cmake
  include(${CMAKE_ROOT}/Modules/CMakeSystemSpecificInformation.cmake)

  # define CMAKE_SHARED_LIBRARY_SUFFIX, CMAKE_SHARED_LIBRARY_PREFIX, etc.
  include(${CMAKE_ROOT}/Modules/Platform/${CMAKE_SYSTEM_NAME}.cmake)
endif()

message(STATUS "CMAKE_SYSTEM:                ${CMAKE_SYSTEM}")
message(STATUS "CMAKE_SYSTEM_NAME:           ${CMAKE_SYSTEM_NAME}")
message(STATUS "CMAKE_SYSTEM_VERSION:        ${CMAKE_SYSTEM_VERSION}")
message(STATUS "CMAKE_SYSTEM_PROCESSOR:      ${CMAKE_SYSTEM_PROCESSOR}")
message(STATUS "CMAKE_HOST_SYSTEM:           ${CMAKE_HOST_SYSTEM}")
message(STATUS "CMAKE_HOST_SYSTEM_NAME:      ${CMAKE_HOST_SYSTEM_NAME}")
message(STATUS "CMAKE_HOST_SYSTEM_VERSION:   ${CMAKE_HOST_SYSTEM_VERSION}")
message(STATUS "CMAKE_HOST_SYSTEM_PROCESSOR: ${CMAKE_HOST_SYSTEM_PROCESSOR}")
message(STATUS "CMAKE_SHARED_LIBRARY_SUFFIX: ${CMAKE_SHARED_LIBRARY_SUFFIX}")
message(STATUS "CMAKE_SHARED_LIBRARY_PREFIX: ${CMAKE_SHARED_LIBRARY_PREFIX}")
message(STATUS "CMAKE_STATIC_LIBRARY_SUFFIX: ${CMAKE_STATIC_LIBRARY_SUFFIX}")
message(STATUS "CMAKE_STATIC_LIBRARY_PREFIX: ${CMAKE_STATIC_LIBRARY_PREFIX}")

Meson C++ standard with fallback

Meson can set a default C++ (or C) langauge standard level with fallback to older versions. This allows recent compilers to support the full functionality of the project code, while falling back for older compilers.

get_option('cpp_std') can be used for logic within meson.build if desired.

project('ffilesystem', ['cpp'],
default_options: ['c_std=c99', 'cpp_std=c++23,c++20,c++17', 'buildtype=release', 'b_ndebug=if-release'])

cpp_std = get_option('cpp_std')

message('C++ standard: ' + cpp_std)

The C / C++ code can use macro feature checks that detect compiler C++ / C standard support.

#if __cplusplus >= 201703L
// C++17 features
#endif

For C code likewise:

#if __STDC_VERSION__ >= 201112L
// C11 features
#endif

Related: CMake C++ standard with fallback