Scientific Computing

It’s worthwhile to upgrade Raspberry Pi OS to have recent Python with system / GPIO libraries. These Python-tied libraries are not always easy to install otherwise. It is generally not worthwhile to upgrade the OS in place. Instead, copy off your files, do a fresh Debian install, and copy your files back. Keep the original SD card in case some setup is hard to figure out.

The best way to upgrade the Raspberry Pi operating system is get a new SD card and load the OS from scratch as described below.

Copy any wanted files from Raspberry Pi to the PC with sftp or rsync. This is in case the original SD card fails.

Setup SD card: the contents of the SD card are erased in the following procedure. Buy a new SD card so that the original SD card remains with a working configuration in case there’s an issue.

Use the Raspberry Pi Imager to select Raspberry Pi OS Lite. BEFORE writing the OS image to SD card, enable SSH server and other desired options.

Headless Raspberry Pi first boot: without a monitor and keyboard on the Pi, the Avahi dæmon on the Raspberry Pi will make it easy to find your Raspberry Pi when you plug into LAN or connect WiFi.

Put the SD card into the Pi, plug an Ethernet cord into the LAN router/switch and from the PC (username, hostname are those picked for the Pi by the Raspberry Pi Imager program):

ssh username@hostname.local

Update Pi software

apt install rpi-update
rpi-update
apt update
apt upgrade

Raspberry Pi Configuration

raspi-config

Raspberry Pi reliable long-term install tips: avoid overclocking the Raspberry Pi for best long-term reliability. Use a heatsink kit for high-stakes remote operations.

Related: completely headless Raspberry Pi setup

Git history can be rewritten with certain caveats. One reason to rewrite history for feature requests is to have project-specified message formatting. Another is to add long comments, denoted by adding at least one blank line beneath the minimal summary message.

To edit messages of prior commits, use git rebase -i and then select “r” or “reword” for the desired commits. This is quicker and easier than “e” or “edit” that is used for editing code or breaking up prior commits.

OpenMPI uses “libz” internally. For the past few years, using OpenMPI from Cygwin also requires:

setup-x86_64.exe -P libopenmpi-devel zlib-devel

Otherwise, when building with OpenMPI with CMake OpenMPI might not be found due to:

Missing MPI_Z_LIBRARY

This might occur on other platforms where libz development libraries are not available when building with libopenmpi.

GNU Make is an important legacy build system. Ninja build system is recommended in general since it’s faster, has more features and generally operates more correctly. Ninja works with CMake and Meson among others.

In general, build systems are useful to:

• build programs much faster (10’s or 100’s of times faster) by:
  • only recompile parts of the code that changed
  • compile / link in parallel
• avoid copy-paste mistakes with excessively long compile commands or piles of Bash scripts

GNU Make parallel builds: by default, GNU Make only uses one thread. For most programs, speeding up compilation is desirable and accomplished via the -j or --jobs option:

make -j

The -j option alone tries to compile all targets in parallel at once. This is typically not desirable or efficient due to excessive context switching. On some systems such as Windows Subsystem for Linux, the bare make -j -l2 may overwhelm the computer because WSL1 doesn’t seem to correctly report the load factor. Thus to do parallel builds with GNU Make on WSL1, manually set the number of parallel build threads perhaps equal to or less than the number of physical cores in your PC. For example an 8 physical core laptop would compile with make -j8.

Caveats: in certain systems like older Raspberry Pi computers, the CPU may go into thermal cutback or suffer undervoltage spikes if using an inadequate power adapter. While the better solution is to provide adequate DC power and CPU cooling, one may choose to compromise by manually specifying the number of threads. For example, with the Raspberry Pi we often use 1 or 2 threads.

make -j1

# or

make -j2

RAM limitation: make -j does not consider system RAM, so your computer may become overwhelmed. Try make -j2 or so until the system memory isn’t over-consumed.

Keeping system responsive: a possible downside of building large programs with make -j is that the computer can become non-responsive to input including mouse and keyboard. A symptom of this is that the Caps Lock keyboard light is very slow to respond. To mitigate the problem automatically, disallow a new thread to start if the system becomes too heavily loaded. This is accomplished with the -l or --max-load option.

Linux system load of 0 to 1 means approximately that no tasks are waiting. So if you want to keep your computer responsible to user input (you can continue using your laptop while big program builds in the background), consider something like -l 1. You can set the -l factor higher to allow make to have increasingly higher priority over other tasks, including user input.

Instead of writing Makefiles, we use CMake or Meson on projects involving compiled code. CMake and Meson each have an integrated parallel-executing test suite, which works very well with continuous integration systems for automated self test. Continuous integration should be a part of all software projects of all sizes, from 50 line programs to operating systems. We recommend Azure Pipelines or GitHub Actions for CI in general.

GNU Make parallel docs

Related: CMake fundamentals

Dealing with text files from many sources, it’s not uncommon to get stray hex codes in the files. These characters may be UTF8 or some other character mapping. They may be invisible or show up as empty squares or other odd glyphs. We had numerous markdown files for this website that had been converted from a legacy blogging system. We observed in certain files invisible characters with hex code C2A0. This is UTF8 for non-breaking space.

Programmatically remove the C2A0 in-place with SED like:

sed -i 's/\xC2\xA0/ /g' myfile.txt

Overwriting Git history requires caution and care. An unlimited amount of work can be permanently lost in doing so. Git repos should have “offsite” backup that is checkpointed–the backup retains more than just the last copy. Overwriting Git history should normally only be done on “feature” branches, not on the “main” branch.

Overwrite Git history: common Git development patterns may require contributors to overwrite feature branch commits to avoid cluttered Git history. This Git development pattern usually has the contributor fork the original repo.

Make a feature branch in the fork, allowing maintainers to edit

git switch -c myfeature

CI pipelines validate/lint contributor pushes

git push -u origin myfeature

If CI errors, require overwriting and force-pushing correction

git commit -am "fixup"

Squash oops/typo commits

git rebase -i HEAD~2 myfeature

The “~2” indicates how far back to allow squashing. To reach farther back in history, increase “2”. Squash the fixup commit(s) by changing “pick” to “f” and then save in the editor that opens automatically.

Once changes are correct, force push.

Related:

• Git fork contribute workflow
• Git manual on rewriting history

CMake has been growing for 20 years, and many major projects have switched to CMake from legacy build systems. Meson can replace CMake or be used alongside CMake. CMake’s long lifetime leaves legacy CMake 2.x scripts with difficult to understand behavior and variable scope. CMake 3.x made considerable modernization, and proposed auxiliary declarative CMake language is being discussed by Kitware.

Meson’s syntax is more declarative than CMake and is non-Turing complete without user functions. Key Meson goals include making meson.build script easier to understand while speeding up the build process itself. Meson vs. CMake is not an either-or choice. Projects can provide both CMake and Meson scripts so that it’s easy for both build systems to be independently used.

pip install meson ninja

Meson most commonly uses the Ninja build backend, which we also recommend for CMake.

A typical new Meson build starts like:

meson build  # from top meson.build directory

meson compile -C build

By default, Ninja builds in parallel.

Convert CMakeLists.txt to meson.build using tools/cmake2meson.py, which makes a first pass at recursively converting CMakeLists.txt to meson.build. The developer will need to manually complete this conversion process, but this script helps eliminate some of the tedious parts.

Python can run Matlab code using GNU Octave via Oct2Py. Python transparently calls Matlab/Octave “.m” functions using GNU Octave instead of Matlab. oct2py uses GNU Octave to run most “.m” code that’s compatible with the GNU Octave version installed. Shared memory (RAM) or disk (temporary file) is used to transfer data between Octave and Python. There are several ways to install GNU Octave

Install oct2py Python ↔ Octave module:

pip install oct2py

Some Octave functions require package install.

If import oct2py does not find Octave or finds the wrong Octave, set the environment variable OCTAVE_EXECUTABLE with the full path to the Octave executable. It’s generally not recommended to add Octave to the system Path on Windows, as that can interfere with MinGW or MSYS2.

Matlab/Octave .m functions are transparently used from Python like:

from oct2py import Oct2Py
oc = Oct2Py()

oc.functionname(arg1,arg2,...)

Python via oct2py can use:

• user functions (".m" files you create)
• builtin functions e.g. svd()
• package functions e.g. signal fir1()

Oct2Py can be greatly sped up by using a RAM drive (tmpfs) instead of the system temporary directory. This may be accomplished by:

from oct2py import Oct2Py
oc = Oct2Py(temp_dir='/run/shm')

oc.functionname(arg1,arg2,...)

Of course, replace /run/shm with your RAM drive location.

Advanced Octave functionality is split off into packages to:

• speed up Octave startup
• enhance stability and development cycles

Thus you’ll see pkg load ... commands where appropriate.

1. create/reuse an .m function with the appropriate input & output variables.
2. call this .m function using Oct2Py from Python

For example, Matlab/Octave fir1() is compared in tests/test_oct2py.py with scipy.signal.firwin().

A simpler Python script example is:

from oct2py import Oct2Py

k=5
p=0.2

with Oct2Py() as oc:
    oc.eval('pkg load signal')
    bmat = oc.fir1(k,p)
print(bmat)
# %%
import scipy.signal

bpy = scipy.signal.firwin(k+1,p)
print(bpy)

For your own .m files, simply call the functions with input/output arguments as in the oc.fir1() line of this example.

Related: call Matlab Engine from Python

Ninja uses GitHub Actions for CI / CD to build and distribute Ninja binaries. If it’s necessary to compile Ninja from source, Ninja is quick to compile by:

git clone https://github.com/ninja-build/ninja

python configure.py --bootstrap

or with CMake:

cmake -B build

cmake --build build

The binary executables for Ninja 1.9.0 may not work on RHEL 7 due to incompatible libc. The symptom of this is like:

ninja: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.21’ not found (required by ninja)

This was a known issue with the Ninja release artifact build process that was fixed.

Workaround: if a current version of Ninja is not available, use Ninja 1.8.2. This may work for other “older” Linux distros.