Build Make external projects from CMake

CMake ExternalProject allows building a wide variety of subprojects isolated from the main CMake project.

Finding GNU Make

For GNU Make Makefile projects, it is necessary to invoke the make command. However, there are several programs named “make” across operating systems. To help ensure the correct GNU Make is selected, we do:

find_program(MAKE_EXECUTABLE
NAMES gmake mingw32-make make
NAMES_PER_DIR
DOC "GNU Make")

Example

A real-life example of CMake with Makefile ExternalProject has multiple Make invocations to build separate Make target groups, where later Make targets depend on the other Make targets being built first. We just show a snippet here for clarity, omitting definition of some of the obvious variables used.

Some key techniques:

  • PATCH_COMMAND: This has a previous configure_file that generates a correct Makefile.inc for the computer + compiler being used
  • CONFIGURE_COMMAND: Make doesn’t have a configure step, so we must define this blank, as otherwise CMake will try to find a CMakeLists.txt in the external project code.
  • BUILD_COMMAND: build the first target(s) that are required by targets in subsequent steps. If there’s no subsequent targets, this is the only build step.
  • INSTALL_COMMAND: We “install” into the project’s build/ hierarchy, as otherwise some external projects have very deep directory structures are are annoying to deal with. Very long paths also cause breakage on Windows.
  • BUILD_BYPRODUCTS: In general we point this at the “installed” files, as otherwise Ninja will error on cmake --build build as Ninja is stricter than Make about the target to source graph
  • ExternalProject_Add_Step: If additional targets need to be built (what in the plain project would be multiple make commands), use Steps to do this in order.
ExternalProject_Add(Scotch
GIT_REPOSITORY https://gitlab.inria.fr/scotch/scotch.git
GIT_TAG v6.1.0
GIT_SHALLOW true
PATCH_COMMAND ${CMAKE_COMMAND} -E copy ${PROJECT_BINARY_DIR}/Makefile.inc ${_src}
CONFIGURE_COMMAND ""
BUILD_COMMAND ${MAKE_EXECUTABLE} -j${Ncpu} -C ${_src} ${_targ}
INSTALL_COMMAND ${MAKE_EXECUTABLE} -j${Ncpu} -C ${_src} install prefix=${PROJECT_BINARY_DIR}
BUILD_BYPRODUCTS ${Scotch_LIBRARIES}  # Ninja needs this
)

ExternalProject_Add_Step(Scotch build_esmumps
COMMAND ${MAKE_EXECUTABLE} -j${Ncpu} -C ${_src}/esmumps install
DEPENDEES build
DEPENDERS install
)

Windows Intel compiler pgopti create full path error fix

When using Intel oneAPI compilers on Windows, you may get an error like:

INTERNAL ERROR: pgopti_Create_Full_Path: buffer too small

This is probably caused by a filepath that’s more than 139 characters. You may not see the paths printed as being that long, because CMake with Ninja does preprocessing on each Fortran file that lengthens the filename.

Fix

The fix is to make the project paths shorter. Users may have to build code from a directory nearer the top of the filesystem hierarchy.

Future-dated blog posts in Hugo

Future-dated blog posts make daily or weekly blogging much more feasible, by allowing one to spontaneously create multiple posts made public over time. Another application is when reviewing embargoed topics, to be ready to deploy on release day. Use caution that the features being reviewed are actually in the release–I’ve seen software drop beta and even release candidate features in the public release.

Hugo

Hugo is perhaps the fastest static website generator, which we prefer over Jekyll. Hugo can handle future-dated blog posts by simply regenerating the web page after the future blog post time has arrived. This can be done with a dummy Git commit and push, or triggering the build via the Netlify API control panel.

The config.yaml option for making future posts appear (which some might use for “pinning” a blog post) is:

buildFuture: true

Omitting that config.yaml option (default: buildFuture: false) allows future blog posts to appear on the desired date upon redeploy after the future post timestamp. To preview future posts on the local computer:

hugo server -F

-F == --buildFuture

set post date

The date / time of a Hugo blog post is typically set with the date: front matter variable. Note the publishDate: and expiryDate: variables for advanced usage. The lastmod: variable for last modified date HTML header can be implemented automatically in config.yaml:

enableGitInfo: true

The last modified date is for updates to a blog post without changing the original posted date of the article. The Git commit update time will thereby be in the article metadata, indicating the “freshness” of the article.

list future and draft posts

Hugo posts can be set as “draft”, normally not rendered for deployment by front matter variable draft: true. Future and draft posts can be listed from the top level Hugo project directory by:

hugo list future

hugo list drafts

Build autotools as CMake ExternalProject

Building an autotools project as a CMake ExternalProject saves the time of converting that other project to CMake. This technique makes it easy to automatically build that other project when it’s not easily installable otherwise, or you wish to build it optimized. This technique has project-specific aspects, but here is a high-level overview. One of the key issues this technique overcomes is the ExternalProject rebuilding each time the main CMake project is rebuilt.

This example is simplified from the p4est-cmake project.

Main CMakeLists.txt

if(NOT p4est_external)
  find_package(p4est)
endif()

if(p4est_external OR NOT p4est_FOUND)
  include(cmake/p4est.cmake)
endif()

cmake/p4est.cmake

set(p4est_external true CACHE BOOL "build p4est library")

include(ExternalProject)

set(p4est_LIBRARIES)
# list each of the library binaries used by the project
foreach(_l p4est sc)
  list(APPEND p4est_LIBRARIES ${PROJECT_BINARY_DIR}/lib/lib${_l}${CMAKE_STATIC_LIBRARY_SUFFIX})
endforeach()

if(EXISTS ${PROJECT_BINARY_DIR}/lib/libp4est${CMAKE_STATIC_LIBRARY_SUFFIX})
  # this is what we use to tell that the ExternalProject was already built,
  # and to then hide the ExternalProject_Add
  set(p4est_FOUND true)
endif()

if(NOT p4est_FOUND)

  ExternalProject_Add(p4est_proj
  GIT_REPOSITORY https://github.com/cburstedde/p4est.git
  GIT_TAG prev3-develop
  PREFIX ${PROJECT_BINARY_DIR}
  CONFIGURE_COMMAND ${PROJECT_BINARY_DIR}/src/p4est_proj/configure
  BUILD_COMMAND make -j
  INSTALL_COMMAND make -j install
  TEST_COMMAND ""
  BUILD_BYPRODUCTS ${PROJECT_BINARY_DIR}/src/p4est_proj-build/src/.libs/libp4est.${CMAKE_STATIC_LIBRARY_SUFFIX}
  )

  ExternalProject_Get_Property(p4est_proj SOURCE_DIR)

  ExternalProject_Add_Step(p4est_proj
    bootstrap
    COMMAND ./bootstrap
    DEPENDEES download
    DEPENDERS configure
    WORKING_DIRECTORY ${SOURCE_DIR})

  file(MAKE_DIRECTORY ${PROJECT_BINARY_DIR}/include)  # avoid race condition
endif()

add_library(p4est::p4est INTERFACE IMPORTED GLOBAL)
target_include_directories(p4est::p4est INTERFACE ${PROJECT_BINARY_DIR}/include)
target_link_libraries(p4est::p4est INTERFACE ${p4est_LIBRARIES})
# set_target_properties didn't work, but target_link_libraries did work

if(NOT p4est_FOUND)
  # this is necessary to trigger the ExternalProject
  add_dependencies(p4est::p4est p4est_proj)
endif()

Switching from autotools to CMake

Switching to CMake from autotools or plain Makefiles gives generally better support of many operating systems, in particular Windows. Another benefit is generally faster speed of configuration and build. In 2020, MonetDB switched from autotools to CMake. They observed MonetDB with CMake builds in 40% of the original time. Note also the improvement due to Ninja instead of GNU Make.

Techniques

I have converted a number of projects from autotools to CMake. Generally, I use an automated autotools to CMake script and then manually compare or redo by inspection of the autotools scripts.

Hugo inline HTML insertion

Hugo shortcodes are a powerful, easy way to template code used in blogs and websites. Ana Ulin describes a one-line Hugo shortcode to inline HTML in Hugo:

Under the top-level Hugo website Git directory, add file “layouts/shortcodes/rawhtml.html” containing:

{{.Inner}}

Then in the Markdown file for the particular blog post, do like (removing the space between the left brace and the left caret):

{{ < rawhtml >}}
<p>arbitrary HTML here</p>
{{ < /rawhtml >}}

example

Git diff single file between branches

Comparing the current Git branch to another Git branch “develop” with a GUI is done like:

git difftool develop

Comparing a single file to another Git branch “develop” is done like:

git difftool develop -- src/myfile.c

Git difftool setup

Meld is a popular tool for comparing files and directories that works on Linux, Mac and Windows. Setup Meld with Git like:

git config --global diff.tool meld
git config --global merge.tool meld