Using CMake to download, verify the checksum of files and extract compressed files is easy and seamless. FetchContent downloads the file at configure time.

include(FetchContent)

set(CMAKE_TLS_VERIFY true)

function(download_file url hash)

FetchContent_Declare(download_${hash}
URL ${url}
URL_HASH SHA256=${hash}
DOWNLOAD_NO_EXTRACT true
)

FetchContent_MakeAvailable(download_${hash})

endfunction(download_file)

# === example
download_file(
  https://upload.wikimedia.org/wikipedia/commons/9/91/Octicons-mark-github.svg
  12794390cce7d0682ffc783c785e4282305684431b30b29ed75c224da24035b4
)

This downloads the file and:

• checks hash signature
• verifies the SSL certificate

CMake uses vendored cURL internally.

Extract compressed file

CMake also can extract compressed files like .zip, .tar.bz2, etc. This command can also specify output directory, extract a subset of files matching a pattern and more.

file(ARCHIVE_EXTRACT INPUT in_file.zip)

Notes

Install latest CMake without sudo using cmake_setup.py.

Related

Meson download file

Patching files with the patch program has been a common task for decades. Due to technical issues with potential patch library implementation, CMake does not include a native patch function. We use CMake to detect if “patch” is available, which it virtually always is on Unix-like systems. For Windows, we use WSL.

The example below is for patching a single file, but could be extended to a directory structure. In this example “my.c” is the source file to be patched, and “my.patch” is the “diff -u” generated patch file.

This guide assumes there is no Internet, no Wifi, no router, no display, no keyboard. It assumes you have a Raspberry Pi and laptop with:

• Ethernet port (built-in or USB-Ethernet adapter)
• plain (or crossover) Ethernet cable–no router/switch is required

Some laptops can also work via USB-C connection and the RNDIS Ethernet driver instead of the Ethernet jack. However, some laptops, particularly on Windows may not work for RNDIS. The symptom of this is that the Pi shows up as a serial port instead of an Ethernet connection. Hence, best to use the Ethernet jack for better reliability.

Before going into the field away from internet, download:

• Raspberry Pi OS Lite .img
• Raspberry Pi imaging tool

Prepare SD card via the Raspberry Pi imaging tool to write the desired SD card image. If the laptop is not on the internet, use the imager tool to write the .img you downloaded previously. After writing the SD card image, in the SD card “boot” partition enable SSH on the headless Pi for first boot by creating an empty file named “ssh”. First navigate in the laptop terminal to this “boot” directory. On Linux or MacOS, this is done by:

touch ssh

On Windows PowerShell, this is done by:

New-Item ssh

N.B.: this is NOT the “boot” directory in the 4 GB main partition of the SD card, this is a separate < 100 MB partition named “boot”. It has other important files like config.txt in it.

Optionally, WiFi is enabled by editing from your laptop on the SD card the file /etc/wpa_supplicant/wpa_supplicant.conf and adding lines like

network={
   ssid="my cool wifi router"
   psk="my wifi password"
}

Optionally, prioritize Wifi networks by the priority field, which can be a positive or negative integer.

• default priority is 0.
• equal priority Wifi is based on signal strength, security, etc.

“eject” or unmount the SD card before removal, to avoid corrupting the SD card file system.

Boot the headless Pi by inserting the SD card into the Pi and powering up the Pi. We use link-local networking, where the IP address will be in the 169.254.*.* range, no DHCP server needed. You may need to manually select this network when you plug the Pi into your laptop. After about one minute, on your laptop plugged directly to the Pi via Ethernet:

ssh pi@raspberrypi.local

default password is raspberry

Logged into the Pi, change the default password to something else with

passwd

Set unique hostname via “raspi-config”. For example, if chose mypi then after Pi reboot:

ssh pi@mypi.local

Set locale to be UTF-8. Assuming you wish to have the United States locale, uncheck the default GB and check en_US.UTF-8 UTF-8. When prompted for default locale, also selected en_US.UTF-8 UTF-8. If you don’t pick a UTF8 locale, Python may give UnicodeDecodeError with UTF8, even when .py script already has # -*- coding: utf-8 -*-.

Again assuming you’re using a United States keyboard, set keyboard layout in raspi-config to Generic 104 keyboard, United States layout to avoid not being able to type symbols properly. Reboot to make the locale settings take effect.

Just a simple network switch connecting Raspberry Pis, PCs and other devices makes an internet-free, pure link-local “off the grid” network. Of course, you can also put the Pis on a wired or wireless network connected to the Internet if desired. The .local address functionality will NOT work over the Internet, but only on the LAN segment your device is on.

Without further configuration, SSH servers listen on all interfaces. Normally this is fine. If you want only specific interface(s) to have the SSH server listen, you will need to research ListenAddress of /etc/ssh/sshd_config and/or IPTables.

Anaconda Python puts itself first on PATH when activated. This can become a problem for libraries like HDF5, where “conda install h5py” puts compiler wrapper “h5cc” on PATH first. Tell CMake to prefer Homebrew for a library like HDF5 by doing:

export HDF5_ROOT=$HOMEBREW_PREFIX

Macports doesn’t have a similar environment variable. We instead hint to CMake by inside the CMakeLists.txt:

# --- detect Macports and hint its location
# this helps avoid issues with Anaconda overriding HDF5 with its broken compiler wrapper
if(APPLE)
  if(NOT DEFINED ENV{HOMEBREW_PREFIX} AND NOT DEFINED ENV{MACPORTS_PREFIX})
    find_program(MACPORTS NAMES port)
    if(MACPORTS)
      cmake_path(GET MACPORTS PARENT_PATH MACPORTS_PREFIX)
    endif()
  endif()
endif()

Checking the hash checksum of downloaded files can indicate if a file has been tampered with. Hash collisions are possible by intentionally manipulating a harmful file to look like the expected file. The simpler the hash function, the more likely hash collisions are–MD5 and SHA1 have demonstrated hash collisions. SHA256 is a popular SHA-2 hash function for which it is hard to generate collisons with today’s computing power.

Observe the SHA256 hash of an empty file:

e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

If encountering this SHA256 hash on a downloaded file, perhaps the download failed in a way that wasn’t otherwise detected, or the server file is indeed empty. Using an “if” statement with this hash can be used to alert users to an empty file downloaded.

Ninja normally deletes each response .rsp file if the target build is successful. When debugging a program, we may need to see all the commands run before failure by keeping the .rsp files.

Ninja keeps the .rsp files after compilation by using option:

ninja -d keeprsp

Ninja is a build backend used by Meson build system and CMake as a fast, modern replacement for GNU Make. .rsp files are simply a plain text file containing the command fragment to be run on the command line.

For example, a build line:

cc -Iinc hello.c -lm

is implemented with an .rsp file by the build system as:

cc hello.rsp

where file foo.rsp contains

-Iinc hello.c -lm

There is currently not a Meson or CMake option to keep the .rsp files. Instead, manually invoke Ninja as above when Meson or CMake has configured the build. That is, instead of:

cmake --build build --target hello -v

or

meson compile -C build

do:

ninja -C build -d keeprsp hello

The full path to executables on the system Path are discovered by Python shutil.which. On Windows, this also auto-adds the .exe extension.

A caveat on Unix-like systems (Linux, MacOS) is that shell aliases are not necessarily found by shutil.which. You should instead append the directory of the desired executable to environment variable PATH.

import shutil

# None if executable not found
exe = shutil.which('ls')

cmd = [exe, '-l']
print(cmd)

Since shutil.which() returns None for non-found executable it is convenient for pytest.mark.skipif

For non-system utilities or other programs not on PATH, where the executable path is known, the path can be specified:

shutil.which('myexe', path=mypath)

Fixed in Python 3.8, but present in earlier Python versions is a bug that requires a string for shutil.which(..., path=str(mypath)).

Intel oneAPI is a useful compiler for CI with the C, C++, and Fortran languages. Even more so than Clang, we find oneAPI compilers give useful debugging output at build and run time. As with GCC and Clang, we typically do CI Debug and Release build and test with oneAPI.

The oneAPI GitHub Actions workflow works with MKL and MPI for C, C++, and Fortran. For simplicity, we didn’t show MacOS, which is also possible.

Sometimes a pull request/merge request needs changes before it can be merged. On GitHub, a typical workflow is like:

git switch -c feature main
git pull https://github.com/friend/repo.git main  # other person's repo branch PR/MR is from

Then one edits the files, and commits the changes. However, if the Git pull command fails like:

fatal: Not possible to fast-forward, aborting.

Then ensuring you’re in the new feature branch:

git pull https://github.com/friend/repo.git main
git reset FETCH_HEAD --hard

Then one edits the files, and commits the changes.

Finally, the generic workflow to merge the PR/MR is:

git switch main
git merge --no-ff feature
git push

If the CI-detected change is git commit --amend or squash and force push over previous commits, the CI may fail with a message like:

fatal: reference is not a tree: <commit id>
The command "git checkout -qf <commit id>" failed and exited with 128 during .

Different CI systems have distinct default Git depth. Checking out all commits wastes time, while too shallow can falsely fail on force push.

We would suggest not specifying the Git clone depth, thereby using the CI system default, unless the repo is very large and Git clone is taking too long. If manually specifying Git depth, the depth must be larger than the number of Git commits your team would ever squash and force push (overwriting prior commits).

For example, in GitHub Actions:

- uses: actions/checkout@v2
  with:
    fetch-depth: 5