List all CMake tests with CTest

As a CMake project grows, increasing complexity can make it hard to discern what tests are to be run and their properties. Perhaps the project logic is unexpectedly omitting necessary tests. The CI system or human can verify the list of tests by:

ctest -N

For machine parsing and human-readable verbose details including fixtures and labels, output JSON:

ctest --show-only=json-v1

To ensure an accurate test list, the project must first be configured and built as usual:

cmake -B build

cmake --build build

ctest --test-dir build -N

CMake file separator

In many cases, using the Unix-type slash file separator / will work even on Windows. Trying to manually specify backslash Windows file separator \ can cause problems in CMake and many other languages. Thus in CMake and other languages like Python, we always use / as path separator, and only transform to backslash for the rare cases it’s needed.

Transform to Unix file separator:

cmake_path(CONVERT "path\in\file.txt" TO_CMAKE_PATH_LIST out)

That switches backslash \ file separators to Unix slash / file separators. This becomes relevant if manually adjusting Include paths by appending to _INCLUDE_DIRS or similar. If backslashes sneak through, all kinds of weird build-time errors can result, and even configure-time errors if you use “check_c_source_compiles()” and similar. As the docs note, put quotes "${mypath}" around the variable expansion to ensure CMake doesn’t mangle the path.

Transform to native file separator is generally more rarely used. CMake can transform paths to native file separator, with the caveat that this can cause unpredictable Windows-specific backslash problems, as with any program.


Related: CMake expanduser ~

Fortran GDB Debugging

Debugging Fortran code with GNU Debugger gdb is roughly akin to debugging from the command line with Python pdb. Intel forked GDB into gdb-oneapi, formerly “gdb-ia”, that replaced “idb”. gdb is capable of far more than what’s listed here, down to stack registers and assembly instructions.

GDB also works with Visual Studio Code, which we generally recommend across coding languages.

Start GDB Fortran debugger: assuming executable myprog with arguments hello and 3:

gdb --args ./myprog hello 3

Run program in gdb (perhaps after setting breakpoints) with

r

A typical debugging task uses breakpoints. Breakpoints are where the debugger stops running until you type

c

Set breakpoints by functionName:lineNumber. Example: Set a breakpoint in function myfun on line 32

b myfun:32

For breakpoints in Fortran modules in this example line 32 of a module named mymod in function myfun:

b mymod::myfun:32

List all scope variables as two separate steps.

  1. local variables
  2. arguments to the function.

Variable type, size (bytes/element), and shape (elements/dim) are available for each variable “var” by

whatis var

Example: a Fortran iso_fortran_env real64 3-D array of size 220 x 23 x 83:

var = REAL(8) (220,23,83)

If “var” is a “derived type” (similar to “struct” in other languages), get the same information about each record (akin to “property”) of the derived type by

whatis var%prop

Local variables are variables used only within the scope of the function–excluding arguments to the function.

info locals

List the names and values of all arguments to the current function:

info args

Example: in integer function myfun(a,b) or subroutine mysub(a,b), upon info args you’d see perhaps

a = 1.5
b = 0.2

If a or b are arrays or structs, array values are printed as well.

Upgrade Anaconda for latest Python

Note: it may be necessary to reinstall Anaconda/Miniconda from scratch if packages break during a Python version upgrade. Consider this before attempting an in-place Python upgrade. There is often a couple month delay between a major Python release and Anaconda defaulting to the new version.

Use the new Python version in a new conda environment by:

conda create -n py3x python=3.x

switch to this environment by

conda activate py3x

For interactive data science GUI applications Jupyter is often recommended. Legacy hard-coded GUIs using external libraries have considerable overhead to maintain, and suffer bit rot far faster than the underlying code. At the very least, be sure your code is usable from the command line and/or as a plain importable module without requiring a graphics library. In stripped-down cloud / embedded applications, unused graphical imports cause stability problems and considerable startup lag.

Related: Why upgrade Python?

Force integer axis labels on Matplotlib

To make a Matlabplot figure axis have integer-only labels, use method like:

ax.yaxis.set_major_locator(MaxNLocator(integer=True))
# or
ax.xaxis.set_major_locator(MaxNLocator(integer=True))

A complete standalone example follows:

import numpy as np
from matplotlib.figure import Figure
from matplotlib.ticker import MaxNLocator

x = np.arange(0.1,10.5,0.1) # arbitrary data

fg = Figure()
ax = fg.gca()
ax.plot(x)

ax.yaxis.set_major_locator(MaxNLocator(integer=True))

fg.savefig("example.png")

If too few ticks are displayed, as per the Matplotlib MaxNLocator, you must have “at least min_n_ticks integers…found within the view limits.” Set “MaxNLocator(nbins=5,integer=True)” or similar if the defaults aren’t forcing integer ticks.

Find source of PyTest warning

PyTest flips on DeprecationWarning and PendingDeprecationWarning as typically those running PyTest are a developer or advanced users. When the package uses warnings.warn to emit a DeprecationWarning, it can be hard to know from where in the tested package a warning is coming from.

To turn on a large amount of warnings, similar to what might be seen on CI:

python -Walways::DeprecationWarning -m pytest

If there is only one type of DeprecationWarning being omitted, a simple way to find the source of the warning is Python warning control -W. This will raise an exception at the warning with traceback:

python -Werror::DeprecationWarning -m pytest

However, often there are multiple DeprecationWarning emitted from different sources, and chances are the one of interest isn’t the first. Perhaps the warning comes from the Python distribution (e.g. Miniconda) itself. In that case, one can insert temporary warning trap into the test function or the user function:

warnings.filterwarnings("error", category=DeprecationWarning)

This might have to be done iteratively to get past the point where the uninteresting DeprecationWarning happen until you home in on the location of the DeprecationWarning of interest.


Related: silence PyTest DeprecationWarning

Git 2.31 mergetool enhancements

Git 2.31 added the ability to maximally resolve the parts of a merge commit, where some parts could not be auto-merged. This workflow isn’t good for some types of project, but a lot of projects and devs do enjoy this setting.

Set this globally (it can also be set per-repo and/or per merge tool):

git config --global mergetool.hideResolved true

Install Matlab in Linux

This procedure works for Matlab (regular and student) on most Linux systems.

Extract all files from the Matlab installer archive, and run (without sudo)

./install

Install under “~/.local/” since it’s tied to your Linux username anyway. Install Symbolic Links to “~/.local/bin/” when asked by the GUI.

Add to ~/.bashrc:

export PATH=$PATH:$HOME/.local/bin

Optional: Add links in your desktop menu


Matlab can be started from Terminal:

matlab

It’s best to run non-interactive Matlab jobs (including CI) like

matlab -batch myscript

NOTE: A GUI is required for normal install

If installing Matlab remotely over SSH, use any one of:


If you cannot click the “click here” for individual site license, try dragging the “click here” to the web browser address bar.


If getting error:

terminate called after throwing an instance of 'std::runtime_error'
  what():  Unable to launch the MATLABWindow application

try removing file “bin/glnxa64/libcrypto.so.1.1” and ./install again.

reference

Install Matlab Engine API in Python

Matlab Engine API allows calling Matlab functions from Python code.

Find the directory Matlab is installed in. From Terminal / PowerShell:

matlab -batch matlabroot

change to the directory indicated then

cd extern/engines/python

Execute these commands (not in Matlab prompt):

  • MacOS or Linux:

    python setup.py build --build-base=$(mktemp -d) install --user
    
  • Windows Powershell:

    python setup.py build --build-base=$env:temp install --user
    

Root / admin permissions are NOT needed.


A simple example of using Matlab from Python:

import matlab.engine
eng = matlab.engine.start_matlab('-nojvm')
y = eng.asin(1.)

eng.quit()

The Matlab Engine should take less than 1 second for Matlab Engine to start when called from Python. Note: some Matlab functions require JVM, thus although much slower to start, you may need to remove the -nojvm option.

Many Matlab numeric classes (single, double, logical) can be converted to Python types like:

numpy.asarray(x)

Python floats pass into Matlab Engine by including a period . after the number.

  • asin(1) fails
  • asin(1.) works

Python can pass N-dimensional arrays to Matlab.

Matlab Engine provides asynchronous call with background=True


References:

Related: