Scientific Computing

Detect if program was compiled with optimizations

Users and developers might accidentally build a program or library without optimizations when they are desired. This could make the runtime 10 to 1000 times or more slower than it would be with optimizations. This could be devastating in computational cost on HPC and cause needless schedule delays. Programmatically detecting or using a heuristic to determine if a program was built with optimizations can help prevent this. Such methods are language-specific.

  • CMake, NDEBUG is set if CMAKE_BUILD_TYPE is Release or RelWithDebInfo.
  • Meson: NDEBUG is set if buildtype is release or debugoptimized with
project(..., default_options: ['b_ndebug=if-release'])

C / C++

There is currently no universal language standard method in C / C++ to determine if optimization was used on build. The presence of macro NDEBUG is used by the standard library to disable assertions. One could use if NDEBUG is defined as an indication if optimizations were used.

bool fs_is_optimized(){
// This is a heuristic, trusting the build system or user to set NDEBUG if optimized.
#if defined(NDEBUG)
  return true;
#else
  return false;
#endif
}

Fortran

If the Fortran code is compiled with preprocessing, a method using NDEBUG as above could be used. Fortran iso_fortran_env provides functions compiler_version and compiler_options. These could be used in a fine-grained, per compiler way to determine if optimizations were used.

Python

Distributed Python environments would virtually always be optimized. One can use heuristic checks to help indicate if the Python executable was built in debug mode. I am not yet aware of a universal method to determine if the CPython executable was built with optimizations.

import sysconfig

debug = bool(sysconfig.get_config_var('Py_DEBUG'))

HDF5 command line tools

HDF5 command line tools h5dump and h5ls are handy to quickly explore HDF5 files from the command line. Backup link to old documentation. They are particularly useful when accessing a remote computer such as HPC where the HDF5 files may be very large and would take a while to transfer to a local computer.


h5ls provides a high-level look at objects in an HDF5 file. Backup link to old documentation. Typically we start examining HDF5 files by printing the dataset hierarchy:

h5ls --recursive my.h5

Determine the filters used (e.g. was the data compressed):

h5ls --verbose my.h5

h5dump can print the entire contents of an HDF5 file to the screen. Backup link to old documentation. This can be overwhelming, so we typically print only the headers to start:

h5dump --header my.h5

Individual variables can be printed like:

h5dump --dataset=myvar my.h5

Determine the filters used (e.g. was the data compressed):

h5dump --properties --header --dataset=myvar my.h5

Related: HDF5 data GUI

C++ size_type property vs size_t

The C++ Standard Library uses size_type as a property of containers like std::vector, std::string, etc. This is generally recommended over using size_t directly.

Example C++ code snippets using size_type property:

std::vector<int> vec;

std::vector<int>::size_type L = vec.size();

//----------------------------------------------
std::string path = "/usr/bin:/usr/local/bin";
constexpr char pathsep = ':';

std::string::size_type start = 0;
std::string::size_type end = path.find_first_of(pathsep, start);

Related: ssize_t for Visual Studio

CI runners - stable vs. updated

CI runners across CI services often update software images regularly, perhaps weekly. This can break workflows, but reflects user devices.

GitHub Actions updates the runners weekly or so. A few times a year on average across projects and operating system this may require updating the CI YaML configuration. Apple updates of XCode a few times a year this can disrupt end users and CI runs.

To have a version stable CI image would generally require private on-premises CI like Jenkins or GitHub Actions for on-premises. Those on-premises CI runners then need maintenance.

The key issue with such frozen CI runners is they are out of date with what end users have. For example, macOS with Homebrew is probably the majority of scientific computing users besides HPC. Homebrew updates often and breaks occur across projects a few times a year. Better to catch that in CI rather than on end user devices.

CMake variables in SCRIPT role

When running CMake standalone scripts like cmake -P script.cmake this is the SCRIPT CMake role. Not all CMake information variables are set in SCRIPT role, in particular, the CMAKE_HOST* and CMAKE_SYSTEM* variables are not set as they are in PROJECT role.

This is a workaround for cmake -P SCRIPT role to get the CMAKE_HOST_* variables. It uses undocumented CMake-internal scripts, but they’ve been present since 2012 and may be unlikely to change.

message(STATUS "CMake ${CMAKE_VERSION}")

get_property(cmake_role GLOBAL PROPERTY CMAKE_ROLE)
if(cmake_role STREQUAL "SCRIPT")
  set(CMAKE_PLATFORM_INFO_DIR ${CMAKE_BINARY_DIR}${CMAKE_FILES_DIRECTORY})
  # define CMAKE_HOST*, CMAKE_SYSTEM*, etc.
  include(${CMAKE_ROOT}/Modules/CMakeDetermineSystem.cmake)
  # set booleans like CYGWIN
  include(${CMAKE_ROOT}/Modules/CMakeSystemSpecificInitialize.cmake)
endif()

message(STATUS "CMAKE_SYSTEM:                ${CMAKE_SYSTEM}")
message(STATUS "CMAKE_SYSTEM_NAME:           ${CMAKE_SYSTEM_NAME}")
message(STATUS "CMAKE_SYSTEM_VERSION:        ${CMAKE_SYSTEM_VERSION}")
message(STATUS "CMAKE_SYSTEM_PROCESSOR:      ${CMAKE_SYSTEM_PROCESSOR}")
message(STATUS "CMAKE_HOST_SYSTEM:           ${CMAKE_HOST_SYSTEM}")
message(STATUS "CMAKE_HOST_SYSTEM_NAME:      ${CMAKE_HOST_SYSTEM_NAME}")
message(STATUS "CMAKE_HOST_SYSTEM_VERSION:   ${CMAKE_HOST_SYSTEM_VERSION}")
message(STATUS "CMAKE_HOST_SYSTEM_PROCESSOR: ${CMAKE_HOST_SYSTEM_PROCESSOR}")

CMake detect if project is top level

CMake can detect if a project is “top level” that is, NOT via FetchContent using PROJECT_IS_TOP_LEVEL and PROJECT_NAME_IS_TOP_LEVEL . For simplicity, we denote these variables in this article as “*_IS_TOP_LEVEL”.

Example use:

if(${PROJECT_NAME}_IS_TOP_LEVEL)
  message(STATUS "${PROJECT_NAME} directly building, not FetchContent")
endif()

For CMake < 3.21:

if(CMAKE_VERSION VERSION_LESS 3.21)
  get_property(not_top DIRECTORY PROPERTY PARENT_DIRECTORY)
  if(not_top)
    set(${PROJECT_NAME}_IS_TOP_LEVEL false)
  else()
    set(${PROJECT_NAME}_IS_TOP_LEVEL true)
  endif()
endif()

Caveats

Directory property PARENT_DIRECTORY and *_IS_TOP_LEVEL are NOT useful for detecting if the child project is being used as an ExternalProject.

These variables are based on the last “project()” command and so are not as universally useful as it first seems. For example, these variables do not work as expected when using ExternalProject. Even setting CMAKE_CACHE_ARGS of ExternalProject does not help, nor does cmake (1) command line options–the CMake-internal setting of *_IS_TOP_LEVEL overrides this attempt to set it. To workaround this, use an arbitrary auxiliary variable to detect if the project is top level.

Example:

Top-level CMakeLists.txt:

ExternalProject_Add(sub1
...
CMAKE_ARGS -DSUB1_IS_TOP:BOOL=false
)

ExternalProject_Add(sub2
...
CMAKE_ARGS -DSUB2_IS_TOP:BOOL=false
)

Subproject CMakeLists.txt

if(DEFINED SUB1_IS_TOP)
  set(SUB1_IS_TOP_LEVEL ${SUB1_IS_TOP})
endif()

Rather than try to directly workaround all the corner cases of *_IS_TOP_LEVEL, using this auxiliary variable allows the user to clearly force the intended behavior. This is useful when the subprojects and main project can build required ExternalProjects, and you want to only build the required ExternalProjects once.

GCC / Clang header clash on macOS

GCC on macOS including Homebrew-installed depends on the macOS SDK. When the macOS SDK is updated, the system headers may become incompatible with GCC versions < 13.3. Specifically, there can be syntax changes requiring C23 but that GCC < 13.3 could not handle.

Homebrew GCC 14.1 and newer work just fine, so the solution is to update GCC.

CMake C++ standard with fallback

CMake can detect features in global property CMAKE_CXX_KNOWN_FEATURES and fallback to a lower C++ standard version if the requested standard is not available. This is useful for building projects that need to support older compilers.

Use feature macros and / or CMake source code checks to define symbols within the code to fallback as necessary.

project(my LANGUAGES CXX)

get_property(cxx_features GLOBAL PROPERTY CMAKE_CXX_KNOWN_FEATURES)

if("cxx_std_20" IN_LIST cxx_features)
  set(cxx_std 20)
else()
  set(cxx_std 17)
endif()

add_executable(my_exe my.cpp)
target_compile_features(my_exe PRIVATE cxx_std_${cxx_std})

CMake 3.28, 3.29 Clang scandep workaround

CMake 3.28.0 .. 3.29.2 have a bug with Clang > 17 if CMAKE_CXX_STANDARD is set to 20 or higher before project() or enable_language(CXX). Specifically, if CMake policy CMP0155 is set to NEW by cmake_minimum_required(VERSION) or otherwise, then CMake 3.28.0 .. 3.29.2 will scan for C++ modules during initial C++ compiler checks, which is not expected or desired. To trivially workaround this issue without otherwise impacting the project or newer CMake versions, do like:

set(CMAKE_CXX_STANDARD 20)
# assuming default settings near top of CMakeLists.txt for readability

# <snip>

if(${PROJECT_NAME}_cxx)  # arbitrary user option

  set(CMAKE_CXX_SCAN_FOR_MODULES OFF)   # workaround CMake 3.28.0 .. 3.29.2 with Clang

  enable_language(CXX)

  set(CMAKE_CXX_SCAN_FOR_MODULES ON)  # optional, if project actually uses C++ modules

endif()

Related: CMake C++ standard with fallback

This issue was fixed in CMake 3.29.3.