Scientific Computing

Building an Autotools project as a CMake ExternalProject saves the time of converting that other project to CMake. This technique makes it easy to automatically build that other project when it’s not easily installable otherwise, or you wish to build it optimized. This technique does not needlessly rebuild the ExternalProject each time the main CMake project is rebuilt.

It is useful to first check that Autotools is available to avoid emitting errors at build time. The goal is to emit errors about missing packages during build system configuration instead of during the build. We do this by including the file FindAutotools.cmake.

INSTALL_COMMAND

Note that “-j” option is NOT used to avoid race conditions in install scripts that might intermittently fail

CONFIGURE_HANDLED_BY_BUILD true

avoid constant reconfigure / rebuild.

“my_LIBRARY” is the known library file(s) built by the Autotools project.

include(GNUInstallDirs)
include(ExternalProject)

set_property(DIRECTORY PROPERTY EP_UPDATE_DISCONNECTED true)
# don't recheck for updates to the Git repo at subsequent CMake reconfigure

set(config_flags)  # parameters desired for ./configure of Autotools

set(my_LIBRARY ${CMAKE_INSTALL_FULL_LIBDIR}/${CMAKE_STATIC_LIBRARY_PREFIX}mylib${CMAKE_STATIC_LIBRARY_SUFFIX})

find_program(MAKE_EXECUTABLE NAMES gmake make mingw32-make REQUIRED)

ExternalProject_Add(mylib
URL https://github.invalid/username/archive.tar.bz2
CONFIGURE_HANDLED_BY_BUILD true
CONFIGURE_COMMAND <SOURCE_DIR>/configure ${config_flags}
BUILD_COMMAND ${MAKE_EXECUTABLE} -j
INSTALL_COMMAND ${MAKE_EXECUTABLE} install
TEST_COMMAND ""
BUILD_BYPRODUCTS ${my_LIBRARY}
)

add_library(mylib::mylib INTERFACE IMPORTED GLOBAL)
target_include_directories(mylib::mylib INTERFACE ${CMAKE_INSTALL_FULL_INCLUDEDIR})
target_link_libraries(mylib::mylib INTERFACE "${my_LIBRARY}")
# need the quotes to expand list
add_dependencies(mylib::mylib mylib)

For Ninja BUILD_BYPRODUCTS is necessary to avoid “ninja: error: “lib” needed by “target”, missing and no known rule to make it”

Some Autotools projects may need a “bootstrap” before “configure”. Add this script if needed:

ExternalProject_Add_Step(mylib
bootstrap
COMMAND <SOURCE_DIR>/bootstrap
DEPENDEES download
DEPENDERS configure
)

CMake ExternalProject allows building a wide variety of subprojects isolated from the main CMake project. For GNU Make Makefile projects, it is necessary to invoke the make command. However, there are several programs named “make” across operating systems. To help ensure the correct GNU Make is selected, we do:

find_program(MAKE_EXECUTABLE
NAMES gmake mingw32-make make
NAMES_PER_DIR
DOC "GNU Make")

A real-life example of CMake with Makefile ExternalProject has multiple Make invocations to build separate Make target groups, where later Make targets depend on the other Make targets being built first. We just show a snippet here for clarity, omitting definition of some of the obvious variables used.

CONFIGURE_COMMAND ""

since Make doesn’t have a configure step, so we must define this blank, as otherwise CMake will try to find a CMakeLists.txt in the external project code.

BUILD_COMMAND

builds the first target(s) that are required by targets in subsequent steps. If there’s no subsequent targets, this is the only build step.

INSTALL_COMMAND

Note that “-j” option is NOT used to avoid race conditions in install scripts that might intermittently fail.

BUILD_BYPRODUCTS

In general we point this at the “installed” files, as otherwise “ninja: error: “lib” needed by “target”, missing and no known rule to make it”. Ninja is stricter than Make about the target to source graph.

include(GNUInstallDirs)

set_property(DIRECTORY PROPERTY EP_UPDATE_DISCONNECTED true)
# don't recheck for updates to the Git repo at subsequent CMake reconfigure

find_program(MAKE_EXECUTABLE NAMES gmake make mingw32-make REQUIRED)

set(my_LIBRARY ${CMAKE_INSTALL_FULL_LIBDIR}/${CMAKE_STATIC_LIBRARY_PREFIX}mylib${CMAKE_STATIC_LIBRARY_SUFFIX})

ExternalProject_Add(mylib
URL https://github.invalid/username/archive.tar.bz2
CONFIGURE_COMMAND ""
BUILD_COMMAND ${MAKE_EXECUTABLE} -j -C <SOURCE_DIR>
INSTALL_COMMAND ${MAKE_EXECUTABLE} -C <SOURCE_DIR> install prefix=${CMAKE_INSTALL_PREFIX}
BUILD_BYPRODUCTS ${my_LIBRARY}
)

CI services typically have a per-job timeout parameter that saves compute resources from being wasted. Overall CI system performance is improved by abandoning jobs taking an unexpectedly long time to run. A typical CI experience is that during busy times, certain tasks like downloading can take much 10x or more longer than usual. A balance is struck between expected task time and timeout limit.

repository install

The apt install or brew install operations that access a remote repository are often run in CI tasks. These tasks might take 10-30 seconds normally, but during heavy CI use times they may take about 10 minutes. This makes setting timeouts less intuitive with jobs that only take a few minutes or less build / test time. Empirically, we find with GitHub Actions that making the overall job timeout 15 minutes or so allows for the sometimes 10 minutes peak download time tolerable.

jobs:

  linux:
    runs-on: ubuntu-latest
    timeout-minutes: 15

    steps:
    - uses: actions/checkout

    - name: install prereqs
      run: |
        sudo apt update
        sudo apt install libopenmpi-dev

    - run: ...

Python pip

We often set a “pip” timeout of 1 or 2 minutes to ensure the CI is using wheels instead of compiling from source, which can take tens of minutes or fail. A job-level timeout can likewise be set.

In this case, rather than setting a job-level timeout-minutes: 6, we know pip for this project takes much less than 1 minute, so we only have to wait one minute if pip goes awry. This becomes important for projects with lots of CI jobs and workflows, you don’t want to have to manually wade through hundreds or thousands of error messages.

jobs:

  linux:
    runs-on: ubuntu-latest
    timeout-minutes: 10

    steps:
    - uses: actions/checkout
    - uses: actions/setup-python
      with:
        python-version: '3.x'

    - run: python -m pip install .[tests]
      timeout-minutes: 1

    - run: pytest
      timeout-minutes: 5

Windows-based devices using RS232 or USB to RS232 connection may work on Linux (including Windows Subsystem for Linux) using WINE. The examples assume a USB to RS232 adapter.

Access serial ports without needing sudo:

adduser $(whoami) dialout

Logout and login again. Plug in USB-RS232 adapter.

ls /dev/ttyUSB*

The USB to serial converter will probably show up as /dev/ttyUSB0 Find the corresponding WINE device port by

ls -l ~/.wine/dosdevices

If necessary, change the serial port mapping in WINE via regedit.

Physical serial ports on Linux such computers with 9-pin / 25-pin RS-232 / RS-485 ports are accessed by:

apt install setserial

setserial -g /dev/ttyS* | grep -v unknown

Related: setup serial port DOSBox-X

CMake requires target parameter WINDOWS_EXPORT_ALL_SYMBOLS set to true to properly use shared libraries on Windows with Visual Studio or Windows Intel oneAPI. oneAPI uses Visual Studio as the backend on Windows. For convenience CMAKE_WINDOWS_EXPORT_ALL_SYMBOLS sets all targets default parameter. Without this parameter, linking fails because the “.lib” import library file isn’t created with the exported shared library symbols.

An example CMakeLists.txt where this parameter is needed to build successfully with Visual Studio or Intel oneAPI on Windows:

set(BUILD_SHARED_LIBS true)

set(CMAKE_WINDOWS_EXPORT_ALL_SYMBOLS true)
# must be before all targets

add_library(a a.c)
add_library(b b.c)

target_link_libraries(b PRIVATE a)

CMake normal variables can be used without ${} evaluation in several contexts including if() statements. Cache and Environment variables do need ${} enclosure even in if() statements to be evaluated.

Example: environment variable “CI” to detect if CMake is running on a CI or not.

# ENV{...} is always false.
# $ENV{...} is correct syntax

if($ENV{CI})
  message(STATUS "CI value: $ENV{CI}")
endif()

When using if() to check for variable existence, then ${} is not used.

if(DEFINED ENV{CI})
  message(STATUS "CI value: $ENV{CI}")
endif()

CMake if() is NOT short circuit. This requires care with undefined variables, which we show by example.

# syntax error if "undef1" is undefined
# if(1 AND ${undef1})
# endif()

if(0 OR undef2)
  message(FATAL_ERROR "unexpected behavior with undefined variable evaluated as string.")
endif()

CMake --debug-find option gives extensive human-readable trace output for where the CMake find_ operations are looking for files. In most CMake projects, there are many find_ operations, and so the cmake --debug-find output can be overwhelming to scroll through. The example commands refer to this CMake script:

# arbitrary other packages not of interest to trace finding
find_package(Git)

# package that's not working right, to debug
find_package(LAPACK)

# arbitrary find_* to trace variable
find_library(MYLIB NAMES mylib)

Package-specific debug-find is available:

cmake --debug-find-pkg=LAPACK

Variable-specific debug-find is available:

cmake --debug-find-var=MYLIB

It’s also possible to target specific find_* calls via CMAKE_FIND_DEBUG_MODE. To debug only the finding of LAPACK:

# arbitrary other packages that are not of interest to trace finding
find_package(Git)

# package that's not working right, to debug
set(CMAKE_FIND_DEBUG_MODE on)
find_package(LAPACK)
set(CMAKE_FIND_DEBUG_MODE off)

This is an informal discussion based on personal experience from late 1990s to present day in the United States during various non-local domestic incidents. Always follow advice from authorities as priority.

The phone tree has become derided as a first choice for organizational awareness. For more informal situations like connecting family groups spread out across states and regions, an informal phone tree can be a backup choice. A key element of this approach is that before a crisis emerges, give/collect phone numbers to several people in a city or region. At least some of these should be people more on the fringes of your social network but still connected in some other way (perhaps a relative in another city of someone you know locally). In this way, the remote and local connections to you are motivated to keep the chain going since there is a friend or family connection beyond yourself. This technique is ideally copied across each ’node’ of the phone tree.

Instead of blast group SMS text messages, which can get delayed and become overwhelming in alerting the recipients with a flood of messages, a phone text tree allows each node to decide which subset is best served by passing on that message. For example, the more maker-savvy folks may appreciate new info on fan filter box construction, while those needing particular supplies may appreciate knowing a store or website suddenly has availability. Blast group text message sent informally even in family groups may suffer from a cascading effect where too much chit chat causes recipients to “tune out” their attention and miss more important messages.

A key idea is to target messaging from each human node using their knowledge of connecting human nodes. This is like a Mechanical Turk for urgent text messaging. The more targeted nature of the messaging incentives spreading more carefully curated information.

General background

Government authorities can broadcast text messages using WEA to cell phones–I have generally had positive experiences with this during severe weather events. The non-WEA, non-CMAS SMS text alert services commonly used by universities and other campuses too often experience delays, where one person may receive an alert minutes after others in the same vicinity.

Lfortran is a unique interactive Fortran compiler working with Jupyter notebook.

conda install lfortran