Five modern, currently-supported compiler families are free-to-use for C, C++ and Fortran.

GCC has broad support of modern standards on a very wide range of computing platforms. GCC’s downside in some cases can be slower runtime performance than compilers having less broad language and platform support.

Intel oneAPI compilers are free to use for any user. The Intel performance libraries like MKL, IPP, and TBB are available at no cost.

LLVM Clang and Flang have significant industry support, including from Nvidia, and are known for high performance.

Nvidia HPC SDK is free to use. A key feature of the HPC SDK compilers is intrinsic support for CUDA Fortran.

IBM OpenXL LLVM-based compilers are currently for POWER CPUs only e.g. ppc64le. IBM OpenXL compilers do not work with a typical x86-based computer.

CMake ExternalProject builds subprojects isolated from the main CMake project. For Visual Studio NMake projects, it is necessary to invoke the nmake command.

find_program(MAKE_EXECUTABLE
NAMES nmake
)

A real-life example of CMake with NMake ExternalProject.

Cray PE can select from multiple compilers as backends for better language standard support while maintaining performance of Cray frontend optimizations. Compilers such as Intel oneAPI themselves use GCC as a backend on Linux. Manual configuration may be required as the default system GCC may be too old for the desired language standard features. Or, the libstdc++ can be too old in the system default GCC. These issues are remediated by purposeful specification of the Cray-Intel-GCC toolchain in a CMake toolchain file.

This toolchain file can be copied to a common location like ~ and used among many projects, such as our cray.cmake.

For CMake project that have ExternalProject inside them, there must be a CMAKE_TOOLCHAIN_FILE parameter to ExternalProject_Add like:

set(cmake_args ...)

if(CMAKE_TOOLCHAIN_FILE)
  list(APPEND cmake_args -DCMAKE_TOOLCHAIN_FILE:FILEPATH=${CMAKE_TOOLCHAIN_FILE})
endif()


ExternalProject_Add(...
CMAKE_ARGS ${cmake_args}
)

The CMake project is configured like:

cmake --toolchain ~/cray.cmake -B build

The Intel oneMKL LAPACK95 library can also be used with Gfortran. Build and install LAPACK95 with our Intel MKL Lapack95 project.

An example CMakeLists.txt, using the FindLAPACK.cmake under cmake/Modules of Netlib LAPACK95

list(APPEND CMAKE_MODULE_PATH "${PROJECT_SOURCE_DIR}/cmake/Modules/")

find_package(LAPACK COMPONENTS LAPACK95 REQUIRED)

add_executable(myexe myprogram.f90)
target_link_libraries(myexe PRIVATE LAPACK::LAPACK95)

Intel oneAPI / oneMKL includes Scalapack. MKL Scalapack library & include files are under the MKLROOT environment variable for Linux and Windows. MKL may be found from CMake like:

find_package(MKL CONFIG REQUIRED)

The exact link flags and compile flags for a particular system are obtained from Intel Link Line Advisor.

The Intel MKL Scalapack examples can be built and run via our example project.

Netlib Scalapack examples can also be used with non-MKL Scalapack.

CMake can detect if a project is “top level” that is, NOT via FetchContent using PROJECT_IS_TOP_LEVEL.

cmake_minimum_required(VERSION 3.21)
project(child)

if(PROJECT_IS_TOP_LEVEL)
  message(STATUS "${PROJECT_NAME} directly building, not FetchContent")
endif()

Note that directory property PARENT_DIRECTORY and PROJECT_IS_TOP_LEVEL properties are NOT useful for detecting if the child project is being used as an ExternalProject.

For CMake < 3.21, “PROJECT_IS_TOP_LEVEL” can be set like:

if(CMAKE_VERSION VERSION_LESS 3.21)
  get_property(not_top DIRECTORY PROPERTY PARENT_DIRECTORY)
  if(NOT not_top)
    set(PROJECT_IS_TOP_LEVEL true)
  endif()
endif()

This CMake function works across Windows, MacOS, and Linux checks that the current CTest script is the only instance of CTest running at the time. We use this in CTest CDash scripts to avoid unexpectedly long-running scripts overlapping with the next day’s runs. That could exhaust memory and crash the computer.

CMake ≥ 3.19 is strongly recommended for general users for more robust and easy syntax. For project developers, we recommend CMake ≥ 3.24 as the new features make debugging CMake and setting up CI considerably easier.

Downloading the latest release of CMake is usually easy. For Linux and Mac, admin/sudo is NOT required.

• Linux: Download/build/install CMake 3 using install_cmake.cmake or zip file
• Mac: brew install cmake or use .dmg
• Windows: use Windows zip file

There is an unofficial PyPI CMake package:

python -m pip install cmake

For platforms where CMake binaries aren’t easily available, use build_cmake.cmake.

CMake 3.24 adds the long-awaited cmake -Bbuild --fresh option to clean the build directory first.

CMake 3.23 further enhances –debug-find to allow debugging all find_* command, or specific find_package or find_* variables. Header files can be more cleanly installed/affiliated with targets, particularly relevant for packaging and installing with target_sources file sets.

CMake 3.22 adds several CMake Environment Variables that are generally useful. CMAKE_BUILD_TYPE default for single configuration build systems. CMAKE_CONFIGURATION_TYPES defaults available configurations for multi-config build systems like Ninja Multi-Config. CMAKE_INSTALL_MODE makes symlinks with copy fallback a good choice for installing programs from CMake. For CTest, the new ENVIRONMENT_MODIFICATION test property makes modifying environment variables for test(s) much easier.

CMake 3.21 adds more preset features, including making “generator” optional–the default CMake behavior will be used to determine generator. The cmake --install-prefix option can be used instead of cmake -DCMAKE_INSTALL_PREFIX=. PROJECT_IS_TOP_LEVEL and <PROJECT-NAME>_IS_TOP_LEVEL identify if a project is at the top of the project hierarchy. ctest --output-junit gives test output in standard tooling format.

CMake 3.20 adds support for Intel NextGen LLVM compiler and NVIDIA HPC compiler. ExternalProject_Add() learned CONFIGURE_HANDLED_BY_BUILD which avoids CMake commanding a reconfigure on each build. try_compiler(WORKING_DIRECTORY) was added. CMake presets in CMakePresets.json now covers configure, build and test, allowing many parameters to be declared with inheritance in JSON. CMake presets are a key feature for CI, as well as user configurations. ctest --test-dir build option avoids the need to manually cd build. cmake_path allows path manipulation and introspection without actually touching the filesystem.

CMake 3.19 added support for ISPC language. string(JSON GET|SET) parsing is very useful to avoid hard-coding parameters. FindPython/find_package accepts version ranges. Intel oneAPI works with CMake ≥ 3.19.6. Emits deprecation warning for cmake_minimum_required VERSION less than 2.8.12. CMakePresets.json enables configure parameter declarations in JSON.

CMake 3.18 adds CMake Profiler

cmake -B build --profiling-output=perf.json --profiling-format=google-trace

Adds REQUIRED parameter to find_program. Adds file(ARCHIVE_CREATE) and file(ARCHIVE_EXTRACT)

CMake 3.17 adds Ninja Multi-Config generator. cmake –debug-find shows what find_*() is doing. Eliminates Windows “sh.exe is on PATH” error. Recognizes that Ninja works with Fortran.

CMake 3.16 adds precompiled headers, unity builds, many advanced project features.

CMake 3.15 adds CMAKE_GENERATOR environment variable that works like global -G option. Enhances Python interpreter finding. Adds cmake --install command instead of “cmake –build build –target install”. Added Zstd compression.

CMake 3.14: I created check_fortran_source_runs(). FetchContent was enhanced with simpler syntax. The transitive link resolution was considerably enhanced in CMake 3.14. Projects just work in CMake ≥ 3.14 that fail at link-time with CMake < 3.14.

We don’t recommend use of the older CMake versions below as they take significantly more effort to support.

CMake 3.13 adds ctest --progress and better Matlab compiler support. Lots of new linking options are added, fixes to Fortran submodule bugs. The very convenient cmake -B build incantation, target_sources() with absolute path are also added. It’s significantly more difficult to use CMake older than 3.13 with medium to large projects.

CMake 3.12 adds transitive library specification (out of same directory) and full Fortran Submodule support. get_property(_test_names DIRECTORY . TESTS) retrieves test names in current directory.

CMake 3.11 allows specify targets initially w/o sources. FetchContent is added, allowing fast hierarchies of CMake and non-CMake projects.

CMake 3.10 added Fortran Flang (LLVM) compiler and extensive MPI features.

CMake 3.9 added further C# and Cuda support, that was originally added in CMake 3.8.

CMake 3.8 added initial Cuda support

CMake 3.7 added comparing ≤ ≥ and version comparisons. Initial Fortran submodule support was added.

CMake 3.6 gave better OpenBLAS support.

CMake 3.5 enhanced FindBoost target with auto Boost prereqs.

CMake 3.4 added if(TEST) to see if a test name exists.

CMake 3.3 added list operations such as IN_LIST.

NOTE: This process is for Intel CPU hardware only! While it may be possible to compile MPICH on Apple Silicon with Intel oneAPI, most programs will not compile or run.

Intel oneAPI for MacOS currently does NOT include MPI. MPICH can be compiled with Intel oneAPI for MPI programs on MacOS with Intel oneAPI. This process also produces “mpiexec” for running MPI programs.

We suggest using the “online installer” download, which is a small download. The “online” installer can be copied over SSH to your HPC user account for example and installed from the Terminal. Install Xcode and Xcode Command Line Tools.

Install the oneAPI Base Toolkit with options:

• Math Kernel Library (oneMKL)
• (optional) GDB debugger

Install the oneAPI HPC toolkit with options:

• Intel C++ compiler
• Intel Fortran compiler

Each time to use Intel oneAPI:

source /opt/intel/oneapi/setvars.sh

MPICH

MPICH will work with Intel oneAPI to provide the MPI libraries and MPIexec with Intel CPU. Download and extract the stable MPICH source. Use commands like:

source /opt/intel/oneapi/setvars.sh

export FC=ifort CC=icc CXX=icpc

mkdir -p $HOME/lib_intel/mpich

./configure --prefix=$HOME/lib_intel/mpich
# the configure step may take tens of minutes

make -j4
# limit number of parallel to avoid crashing build

make install -j4

To use this MPICH with Intel oneAPI in general via CMake, make a file ~/intel.sh like:

source /opt/intel/oneapi/setvars.sh

export FC=ifort CC=icc CXX=icpc

export MPI_ROOT=$HOME/lib_intel/mpich

Troubleshooting

If upon compiling a C program stdio.h isn’t found or a C++ program wchar.h isn’t found, install Xcode Command Line Tools for the Xcode version currently installed. If the program persists, the problem may be that the Xcode include path may not be in Intel’s scripts. To workaround this issue, look for the Command Line Tools include directory and make a file ~/intel.sh including (check the directory name):

#!/bin/sh

set -o errexit

source /opt/intel/oneapi/setvars.sh

export LIBRARY_PATH=$LIBRARY_PATH:/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/lib
export CPLUS_INCLUDE_PATH=/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include

export CXXFLAGS=-I$CPLUS_INCLUDE_PATH
export CFLAGS=$CXXFLAGS

export FC=ifort CC=icc CXX=icpc
# oneAPI on MacOS doesn't yet have the IntelLLVM compilers

and

source ~/intel.sh

to use Intel oneAPI.

If you try to use Apple Silicon with Intel oneAPI:

Intel MKL FATAL ERROR: This system does not meet the minimum requirements for use of the Intel(R) Math Kernel Library.
The processor must support the Intel(R) Supplemental Streaming SIMD Extensions 3 (Intel(R) SSSE3) instructions.
The processor must support the Intel(R) Streaming SIMD Extensions 4.2 (Intel(R) SSE4.2) instructions.
The processor must support the Intel(R) Advanced Vector Extensions (Intel(R) AVX) instructions.

On Windows, when building an executable target in Debug mode using Visual Studio or Intel oneAPI, it may be necessary to also have the libraries linked by the target to have Debug symbols. For example, when building in CMake:

cmake -B build -DCMAKE_BUILD_TYPE=Debug

cmake --build build --config Debug

This may be indicated by messages like:

error LNK2038: mismatch detected for '_ITERATOR_DEBUG_LEVEL': value '0' doesn't match value '2' in main.cpp.obj

error LNK2038: mismatch detected for 'RuntimeLibrary': value 'MD_DynamicRelease' doesn't match value 'MDd_DynamicDebug' in main.cpp.obj

The solution is to build the libraries with Debug mode as well.s