CMake find_program script

CMake find_program does not generally consider NAMES parameter to have file suffixes unless manually specified. A special case is Windows, where .com and .exe file suffixes are also considered.

If on Windows and an executable “hello.exe” or “hello.com” exists, then CMake will find it. CMake would NOT find “hello.exe” on non-Windows platforms, where no file suffix is expected.

NOTE: CMAKE_EXECUTABLE_SUFFIX ONLY affects find_program() in CMake role PROJECT

find_program(v NAMES hello)

Shell scripts of any file suffix on any operating system are found iff:

  1. script file executable flag is set (find_program general requirement)
  2. script file suffix is specified as part of NAMES parameter

A complete standalone example:

Intel oneAPI icpx Windows

Intel oneAPI 2023 provides GNU-like and MSVC-like compiler drivers for Windows. This is a good thing, but CMake had assumptions that only C++ compiler driver icpx OR icx would be present. CMake 3.25.2 fixed oneAPI 2023 compiler detection on Windows. Long term, GNU-like support for Windows may also come. Until you update CMake, you can workaround this issue of C++ oneAPI compiler detection on Windows by configuring the CMake project like:

cmake -Bbuild -DCMAKE_CXX_COMPILER=icx

Disable Visual Studio debug model window

Visual Studio executables built in Debug mode by default pop up modal debug windows if an unhandled exception occurs. This can be annoying to developers particularly when unit testing a project. On remote systems, modal windows can become a real issue if the modal window is accidentally off-screen. In such cases it is sometimes hard to get the modal window back to the main desktop to be closed.

Adding a few lines of code to the C++ program works around this issue by redirecting the error text to stderr console and not popping up the modal window. _CrtSetReportMode keeps the model window from appearing. _CrtSetReportFile redirects the message text to stderr so that the message can be diagnosed.

#ifdef _MSC_VER
#include <crtdbg.h>
#endif

int main(){
#ifdef _MSC_VER
    _CrtSetReportMode(_CRT_ASSERT, _CRTDBG_MODE_FILE);
    _CrtSetReportFile(_CRT_ASSERT, _CRTDBG_FILE_STDERR);
    _CrtSetReportMode(_CRT_WARN, _CRTDBG_MODE_FILE);
    _CrtSetReportFile(_CRT_WARN, _CRTDBG_FILE_STDERR);
    _CrtSetReportMode(_CRT_ERROR, _CRTDBG_MODE_FILE);
    _CrtSetReportFile(_CRT_ERROR, _CRTDBG_FILE_STDERR);
#endif

// rest of program
}

CTest WILL_FAIL segfault

By design, CTest WILL_FAIL test property does not handle segmentation faults and other unexpected terminations. That is, a failing test will also show as failed in CTest, when the desire may have been to show an intended failure as success via WILL_FAIL. A simple workaround for this is to make a wrapper script that calls the intentionally segfaulting executable and catch that result, and pass that to CTest, as in this example project.

Visual Studio memory leak detection

Visual Studio can detect memory leaks in programs with _CrtDumpMemoryLeaks. This minimal example doesn’t do the printout. Using these checks requires the project is build in Debug mode.

More complete working minimal example that prints the memory diagnostics with Visual Studio. On Linux, Valgrind can be used to detect memory leaks. Numerous other free memory checkers are available and work with CMake CTest frontend.

#ifdef _MSC_VER
#include <crtdbg.h>
#endif

int main(void){
char* c

c = malloc( 100 );
// unfreed memory, a deliberate leak

// near the end of the function to be checked
#ifdef _MSC_VER
  _CrtDumpMemoryLeaks();
#endif

return 0;
}

Python distutils removal

Python distutils is deprecated in PEP632. Setuptools 49.1.2 vendored distutils, but has experienced some friction in setuptools 50.x since so many packages monkeypatch distutils due to the little maintained status of distutils for several years.

With distutils proposed deprecation in Python 3.10, migration to setuptools is a topic being worked on by major packages such as Numpy. Aside from major packages in the Scipy/Numpy stack, I don’t recall many current packages setup.py relying on distutils. However, there is code in some packages using import distutils that could break.

I applaud the decision to remove distutils from Python stdlib despite the fallout. The fallout is a symptom of the legacy baggage of Python’s packaging. Non-stdlib packages like setuptools are so much more nimble that sorely needed improvements can be made more rapidly.

Reference: bug report leading to PEP632

Matplotlib AVI / MP4 movie

Matplotlib on any platform can use FFmpeg, Avconv or Mencoder to directly write lossy or lossless compressed movies created from sequences of plots.

Instead of creating hundreds of PNGs, or skipping plots and missing details, Matplotlib movies of a large sequence of plots is highly effective for many processes that evolve across time and/or space.

Alternatively, convert a stack of PNGs to AVI. It’s simpler and often faster and more robust to use Matplotlib.animation.

Lossy

Quality: the default auto bitrate makes excessively compressed, blocky movies. Override the default auto-bitrate with the following snippet:

import matplotlib.animation as anim

Writer = anim.writers['ffmpeg']
writer = Writer(fps=15, codec='mpeg4', bitrate=1e6)
#
with writer.saving(fg, fn,100):
    # code to plot/update figure
    writer.grab_frame(facecolor='k')

Lossless

In matplotlib_writeavi.py, just four added lines of code do the AVI writing. First line tells Matplotlib to use FFmpeg. Second line tells Matplotlib to make a lossless FFV1 video at 15 frames/sec. One can optionally use codec='mpeg4', but lossy encoding can wash out details of plots. Third line says to use 100 DPI (smaller DPI–smaller file and movie size).

import matplotlib.animation as anim

#...

Writer = anim.writers['ffmpeg']
writer = Writer(fps=15, codec='ffv1')
# ...
with writer.saving(fg, fn, 100):
# ...
   writer.grab_frame(facecolor='k')

Troubleshooting

For problems playing back the .avi file, try omitting the codec='ffv1' parameter.

Minimum AVI frame rate: less than 3 fps can invoke bugs on VLC. VLC has trouble with slow frame rate video from any source.

macOS terminal SSH locale

The macOS terminal defaults to UTF8. When SSHing into a Mac computer from a non-macOS computer, or any computer with a different locale, there may be problems running programs on the remote Mac where locale is important. For example, a Linux system with “C” locale may cause .zip archive extraction on the remote Mac to fail like:

Pathname cannot be converted from UTF-8 to current locale.

Locally on the Mac (or using Remote Desktop over SSH), check locale with:

% locale

LANG="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_CTYPE="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_ALL=

while from a Windows or Linux SSHing into the Mac may result in:

% locale

LANG=""
LC_COLLATE="C"
LC_CTYPE="C"
LC_MESSAGES="C"
LC_MONETARY="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_ALL=

Fix

We resolved this issue by creating on the remote Mac a file “locale.sh” containing:

export LANG="en_US.UTF-8"
export LC_COLLATE="en_US.UTF-8"
export LC_CTYPE="en_US.UTF-8"
export LC_MESSAGES="en_US.UTF-8"
export LC_MONETARY="en_US.UTF-8"
export LC_NUMERIC="en_US.UTF-8"
export LC_TIME="en_US.UTF-8"

then run one time when needed:

source ~/locale.sh

This fixed an issue we had with CMake not extracting a .zip file for ExternalProject URL with the error noted at the top of this page.

Another workaround as noted above is to use Remote Desktop over SSH.

Python minimal package with pyproject.toml

Python packaging can be described in pyproject.toml alone per PEP 621. These packages are installable in live developer mode:

python -m pip install -e .

Or via PyPI like any other Python package. It can be most effective to put all project configuration, including Python package prerequisites in pyproject.toml alone as a single source of truth. pyproject.toml is human-readable and machine-parseable without first installing the package. Putting all package metadata into pyproject.toml instead of setup.py gives benefits including:

  • reproducible results
  • security risk mitigation
  • dynamic prerequisite tree based on Python version etc.
  • static or dynamic package version

This is an example of a minimal pyproject.toml that works all alone, no other metadata files required, except perhaps MANIFEST.in for advanced cases. The __version__ is contained in file mypkg/__init__.py as Python code:

__version__ = "1.2.3"

pyproject.toml:

[build-system]
requires = ["setuptools>=61.0.0", "wheel"]
build-backend = "setuptools.build_meta"

[project]
name = "mypkg"
description = "My really awesome package."
keywords = ["random", "cool"]
classifiers = ["Development Status :: 5 - Production/Stable",
 "Environment :: Console",
 "Intended Audience :: Science/Research",
 "Operating System :: OS Independent",
 "Programming Language :: Python :: 3",
]
requires-python = ">=3.7"
dynamic = ["version", "readme"]

[tool.setuptools.dynamic]
readme = {file = ["README.md"], content-type = "text/markdown"}
version = {attr = "mypkg.__version__"}

PEP8 checking via flake8 is configured in .flake8:

[flake8]
max-line-length = 100
exclude = .git,__pycache__,doc/,docs/,build/,dist/,archive/
per-file-ignores =
  __init__.py:F401

MANIFEST.in is used to specify external files installed.

Classifiers are optional and help projects indexing in PyPI and search engines. Classifiers must be from the official classifier trove or they will fail when uploading a package to PyPI.

Python can easily import Fortran code using f2py. See this f2py example setup.py.

Git LFS disable

If one is not using Git LFS but has a system with Git LFS installed, it can be useful to uninstall Git LFS to avoid problems like:

Remote "origin" does not support the Git LFS locking API. Consider disabling it with:
  $ git config lfs.locksverify false

To remove Git LFS, simply type:

git lfs uninstall