Upgrade Anaconda for latest Python

Note: it may be necessary to reinstall Anaconda/Miniconda from scratch if packages break during a Python version upgrade. Consider this before attempting an in-place Python upgrade. There is often a couple month delay between a major Python release (e.g. Python 3.7 to Python 3.8) and Anaconda defaulting to this version. We strongly suggest waiting until the new Python version is the default before trying to upgrade, as many packages many be missing or broken until then. A good example of why to wait for default Anaconda Python is Tensorflow that did not yet work with Python 3.8 as of March 11, 2020.

Given the caveats above, if appropriate it is possible to upgrade to Python 3.8 by

conda create -n py38 python=3.8

switch to this environment by

conda activate py38

GUI advice

For interactive data science GUI applications Jupyter is often used. Legacy hard-coded GUIs using external libraries have considerable overhead to maintain, and suffer bit rot far faster than the underlying code. At the very least, be sure your code is usable from the command line and/or as a plain importable module without requiring a graphics library. In stripped-down cloud / embedded applications, unused graphical imports cause stability problems and considerable startup lag.

Matlab dependency graph

Matlab can generate dependency graphs and export to GraphML files. The GUI dependency analyzer also shows missing variable and function definitions for the whole project in one view. The implicit static analysis of the dependency analyzer seems to be more thorough than the command requiredFilesAndProducts.

Even if you don’t typically use the Matlab “Project” feature, it’s useful for quality assurance to run this tool on Matlab code to find hidden problems that escape the imperfect nature of unit tests. Regardless of coding language, we use a layered approach for code quality assurance including static analysis, unit tests and integration tests.

Windows Git Credential Manager

Git over SSH can use SSH-agent to remember SSH keys for Linux and Windows. For those not using Git over SSH but tiring of typing the Git password, on Windows the Git Credential Manager can fix that situation. The install is small and quick, and to use Git Credential Manager, be sure via

git remote -v

that the connection is https:// and not ssh:// where appropriate. Upon trying to git push or git pull, the Git Credential Manager will appear. It works with Two Factor Authentication as well, storing in Windows Credential Manager. The keys can be deleted from Windows Credential Manager if needed, and / or the Personal Access Token automatically generated can be deleted from GitHub.

Notes

If using automatic Git push SSH that would need to be disabled to allow git push to use https so that Git Credential Manager is invoked.

SSH-agent on Linux, WSL and Windows

SSH-agent remembers SSH Public Key authentication, which can be time-limited by the user. This avoids the user having to type the password for each SSH connection, especially relevant to using Git over SSH. Native Windows has SSH including SSH-agent, and separately WSL also can use SSH-agent. SSH-agent works well with Git over SSH.

Add SSH keys to SSH-agent

To use SSH-agent, add SSH keys like:

ssh-add -t 30m ~/.ssh/mykey
-t 30m
remember authentication for a period of time (here, 30 minutes)

One can optionally remove all SSH-agent keys from RAM by

ssh-add -D

Note that if the SSH private key was manually deleted, access to the remote SSH server is lost until a new private key is placed on the remote server when an SSH key is removed from SSH-agent.

Enable SSH-agent

Each operating system has a distinct method of enabling SSH-agent, as follows.

Windows

Windows SSH-agent is off by default, but can be enabled from PowerShell:

Start-Service ssh-agent

The status of Windows SSH-agent can be checked from PowerShell:

Get-Service ssh-agent

if status is “Running” then SSH-agent should be working.

Linux / WSL

This works for Linux in general, including Windows Subsystem for Linux.

Add to ~/.bashrc:

if [ -z "$(pgrep ssh-agent)" ]; then
   rm -rf /tmp/ssh-*
   eval $(ssh-agent -s) > /dev/null
else
   export SSH_AGENT_PID=$(pgrep ssh-agent)
   export SSH_AUTH_SOCK=$(find /tmp/ssh-* -name agent.*)
fi

Notes

CMake specifying generator full path

Normally it is not necessary to specify the path to the CMake generator backend, assuming the generator executable is on the system PATH. In some cases, perhaps testing CMake with several different versions of a generator, one may wish to specify the generator executable via CMAKE_MAKE_PROGRAM. When doing so, the absolute path to the generator is necessary or CMake will not find it.

For example, suppose on a CI one extracts Ninja 1.10 to use CMake and Ninja on a Fortran project. With GitHub Actions on a Linux image, this would be specified like:

    - name: Install Ninja
      run: |
        wget https://github.com/ninja-build/ninja/releases/download/v1.10.0/ninja-linux.zip
        unzip ninja-linux.zip

    - name: CMake configure
      run: cmake -G Ninja -DCMAKE_MAKE_PROGRAM=$(realpath ./ninja) -B build

The key point is that we specify the full path from a relative path with realpath to Ninja (when necessary) like:

cmake -G Ninja -DCMAKE_MAKE_PROGRAM=$(realpath ./ninja)

Using CMAKE_ROLE to detect CMake run mode

The operating mode of CMake can be detected via CMAKE_ROLE. This variable must be read with:

get_property(cmake_role GLOBAL PROPERTY CMAKE_ROLE)

Here, the variables cmake_role is created and can then be used in if() statements as needed.

Detect WSL from Python in Windows

It’s trivial to detect if one is running inside WSL. It’s also straightforward to detect if WSL is available and working on a Windows PC. These functions are using only Python standard library modules.

Detect if inside WSL

This function detects if Python is running in WSL.

import uname


def in_wsl() -> bool:
    """
    WSL is thought to be the only common Linux kernel with Microsoft in the name.
    """

    return 'Microsoft' in uname().release

Detect WSL from Windows

import os
import shutil


def wsl_available() -> bool:
    """
    heuristic to detect if Windows Subsystem for Linux is available.

    Uses presence of /etc/os-release in the WSL image to say Linux is there.
    This is a de facto file standard across Linux distros.
    """
    if os.name == "nt":
        wsl = shutil.which("wsl")
        if not wsl:
            return False
        # can't read this file or test with
        # pathlib.Path('//wsl$/Ubuntu/etc/os-release').
        # A Python limitation?
        ret = subprocess.run(["wsl", "test", "-f", "/etc/os-release"])
        return ret.returncode == 0

    return False

Ignore Meson subprojects with .gitignore

Meson projects using Meson subprojects have a project directory structure like:

meson.build
main.c
subprojects/
  lapack/
  lapack.wrap

where “subprojects/lapack/” is automatically created as a Meson subproject from the metadata in “subprojects/lapack.wrap”. Using the negate .gitignore syntax Git will ignore subdirectories but track the subprojects/*.wrap files:

include in the project .gitignore:

subprojects/

!subprojects/*.wrap

This technique can be applied to similar non-Meson scenarios.

CMake Ninja Multi-Config generator

We generally recommend using Ninja ≥ 1.10 with CMake ≥ 3.15, especially for large projects (including Fortran) to speed up rebuild times significantly and avoid erratic problems with slower GNU Make on large projects. CMake 3.17 added

cmake -G "Ninja Multi-Config" -B build

which allows building Debug and Release builds or even cross builds without regenerating build*.ninja files for each build type.

Ninja Multi-Config generator options may be used transparently with older CMake and different generators. The default CMake generator can be set with environment variable:

CMAKE_GENERATOR="Ninja Multi-Config"

To make the default multi-config build “Release” while making Debug config also available, add to CMakeLists.txt:

set(CMAKE_CONFIGURATION_TYPES "Release;RelWithDebInfo;Debug" CACHE STRING "Build type selections" FORCE)

Because Release is listed first, the default build type is Release. With those settings, one can quickly build Debug and Release and test like:

cd build

# Release default per above settings
cmake --build .

ctest -C Release

Then update and test Debug by:

cmake --build . --config Debug

ctest -C Debug