Google Shared Drives benefits and caveats

Google Shared Drive is for groups of people with Gmail addresses (corporate or personal). Public folder or limited access link-sharing of folders and files in Shared Drives are possible as with personal Google Drive. Moving files is done in bundles of files, not by folder. Google Drive for desktop works for Shared Drives like individual Google Drive for read/write access. Google Shared Drives and personal Google Drive can be synced with rclone, which is especially handy for HPC and cloud computing. If using Windows Subsystem for Linux, keep the files on the native Windows filesystem.

Fortran build systems

Two of the most important aspects of a build system are generation speed and build/rebuild speed. For non-trivial projects in any compiled language, build speed can be a significant pain point. Build speed is the time it takes to first build project targets. Rebuild speed is the time taken on subsequent builds of a target, where most build artifacts can be reused, but the dependencies and sources must be scanned for changes since last build.

Even for medium projects, the seconds lost rebuilding add up lost productivity for developers. CMake and Meson are among the fastest generators, while SCons and Autotools are among the slowest. CMake and Meson have full support for Ninja. Ninja is generally significantly faster than GNU Make, particularly for the case where only a few out of very many files need to be rebuilt.

Meson is a Python program without external dependencies. It’s easy to run the latest development snapshot or track Git changes as a result.

git clone https://github.com/mesonbuild/meson
cd meson
pip install -e .

Since Meson generates Ninja build files, you will need to download Ninja. With Meson, the code build process goes like:

meson build

meson compile -C build

CMake has existed since the late 1990’s, and has broad usage. Build a CMake project like:

cmake -B build

cmake --build build --parallel

Trendnet TEG-s80g vs. TEG-s82g

The Trendnet TEG-S80g version 3.0 - 4.1 lack LED speed indicators, with only active (plugin detection) indicators for each port.

Model/RevLED speed indicatormetal jacksbuffer memory (kB)MAC address table (entries)
TEG-S80g Rev 1.0yesyes1284K
TEG-S80g Rev 2.1yesyes1288K
TEG-S80g Rev 3.0nono2568K
TEG-S80g Rev 4.1nono1924K
TEG-S82g Rev 2.0nono2568K

The TRENDnet TEG-S80g model has been reliable for me across a decade of use in small instrument networks with challenging physical environments. I have had them survive room flooding, where water drained from the ceiling onto a powered-on TEG-S80g. Upon disassembly there was hardly any moisture inside, and the TEG-S80g continues to function.

Trendnet internals

WSL mount external storage and network drives

Windows Subsystem for Linux can use (mount) external storage including: SD card, USB thumb flash drives, CD / DVD drives (CDFS), network drives, and UNC paths. Coming soon: new wsl --mount syntax that will extend WSL connectivity to Linux filesystems.

WSL can mount and use hardware external drives formatted as FAT, ExFAT or NTFS. If Windows changes the external drive letter on a subsequent session, you need to repeat this process. The commands are typed into the Windows Subsystem for Linux Terminal.

Create a mount location in WSL and mount the drive (assume the drive shows in Windows as “F:"):

mkdir /mnt/f

mount -t drvfs f: /mnt/f

One can then create and manipulate files from both Windows and WSL on the same external drive.


For network storage, assume networked storage is already showing in Windows under \\server\share and we want to access this network storage from WSL as /mnt/share.

Create a mount location in WSL and mount the network share:

mkdir /mnt/share

mount -t drvfs '\\server\share' /mnt/share

Reference: external drive mount on WSL

MyPy PEP 585, 604 support

MyPy 0.800 added support for PEP 585 and 604, which bring the new concise Python 3.10 type annotation syntax to earlier Python versions. The new type annotation syntax works all supported Python versions, if each file using them has at the top:

from __future__ import annotations

Separately, Numpy 1.20 made the long-awaited Numpy type hinting a reality. As too often the case, I became aware of these changes by newly failing CI cases. I was able to update syntax fairly quickly to take benefit of these new type annotation features.

MyPy Python static type hinting quick start

The benefits of Python static type checking and examples have been discussed at length and widely adopted and funded by major tech companies, especially Dropbox. Python static type checking enhances code quality now and in the future by defining (constraining) variables and functions (methods). Type hinting is compatible with Python 2 when the correct syntax is used. MyPy can check type hinting in Python 2 mode.

Type enforcement can be done with assert. Type hinting is more concise, flexible and readable than assert, with significantly less performance impact. Type hinting is being continually enhanced in CPython, numerous IDEs and type annotation checkers. With type hinting, the hint is right at the variable name (e.g. in the function declaration), while assert must occur in the code body.

MyPy is installed and upgraded by:

pip install -U mypy

MyPy static type checker considers the following to be interchangeable (valid) due to duck typing:

  • intfloat
  • floatcomplex

Note that str is not equivalent to bytes.

Usage

Create a top-level project file “.mypy.ini” containing at least

[mypy]

files = src/

assuming Python package files are under “src/” Then issue command:

mypy

Note this command checks the package and not the top-level scripts, which must be manually specified. Configure file .mypy.ini to eliminate nuisance errors or otherwise configure mypy.

It takes a little practice to understand the messages. Where multiple types are accepted, for example, str and pathlib.Path use typing.Union. See the examples below.

Examples

Many times a function argument can handle more than one type. This is handled as follows:

from __future__ import annotations
from pathlib import Path


def reader(fn: Path | str) -> str:
    fn = Path(fn).expanduser()

    txt = fn.read_text()

    return txt

Another case is where lists or tuples are used, the types within can be checked (optionally):

from __future__ import annotations


def reader(fn: Path | str) -> tuple[float, float]:
    fn = Path(fn).expanduser()

    txt: list[str] = fn.read_text().split(',')

    latlon = (float(txt[0]), float(txt[1]))

    return latlon

Or perhaps dictionaries, where optionally types within can be checked:

from __future__ import annotations


def reader(fn: Path | str) -> dict[str, float]:
    fn = Path(fn).expanduser()

    txt: list[str] = fn.read_text().split(',')

    params = {'lat': float(txt[0]),
              'lon': float(txt[1])}

    return params

If many value types are in the dictionary, or possibly some types are not yet supported for type hinting, simply use typing.Any e.g.

dict[str, typing.Any]

The default where no type is declared is typing.Any, which basically means “don’t check this variable at this location in the code”.


As in C++, Python can type hint that a function must not return.

def hello() -> typing.NoReturn:
    print("hello")
error: Implicit return in function which does not return

This is used for functions that always raise an error or always exit, and in general to help ensure control flow is not returned.

GUI viewers for HDF5 / NetCDF4 data

HDF5 is a popular data container format, a filesystem with a file. While the many programs supporting HDF5 like Matlab can manually plot data, it is useful to have a standalone simple data browser.

The canonical HDF5 / NetCDF4 / FITS capable HDFview is maintained by the HDF Group. HDFview enables editing (writing) as well as reading HDF5. MacOS, Windows and Linux users can simply download the HDFview binaries. Linux users can alternatively do like apt install hdfview, but be aware some distros have broken HDFview. If the Linux distro HDFView is broken, simply download the CentOS HDFView binary–it will work on Ubuntu / Debian as well.


The PyTables group maintains ViTables is a Python-based HDF5 viewing program based on PyQT5. PyTables may be installed liek:

pip install pyqt5 vitables

The Java program PanoplyJ is available for MacOS, Linux and Windows. PanoplyJ does not require compiling, just run it using Java. Linux PanoplyJ requires Java 8 or newer, and can also work with Java 9.

apt install openjdk-8-jre

Extract PanoplyJ and make executable:

unzip PanoplyJ*.zip -d ~

chmod +x ~/PanoplyJ/panoply.sh

Run PanoplyJ by:

~/PanoplyJ/panoply.sh

Make a command-line shortcut by adding to ~/.bash_aliases

alias panoply='$HOME/PanoplyJ/panoply.sh'

If you get:

Unknown superblock version=3

You may have an older version of PanoplyJ whose NetCDF-Java library isn’t ready for the newer HDF5 file version. If writing the HDF5 files from h5py use libver=earliest to use the oldest possible HDF5 file API:

with h5py.File('myfile.h5', libver='earliest') as f:
   ....

Check HDF5 files for corruption

Use modern compilers in Red Hat

Often we need a more modern compiler than is included in Red Hat for HPC systems. Once an admin sets up “devtoolset” they can be loaded by any user thereafter. A common devtoolset use is for gfortran:

yum install centos-release-scl

yum install devtoolset-8-gcc devtoolset-8-gcc-gfortran

Thereafter, users without sudo use the new gfortran version by:

scl enable devtoolset-8 bash

Unison file synchronizer

Unison file synchronizer works like a mashup of Dropbox, rsync and Git, although internally working via its own means. Unison attempts to remotely sync changes in files between a local and (typically) remote location. We have used Unison to manage systems on isolated remote networks. That is, where the remote network is isolated from directly accessing the Internet and so Dropbox etc. aren’t available.

Assuming one trusts Unison, this could be useful for cases where it is desirable to keep files out of the “cloud”. Unison file synchronizer has executable downloads for popular operating systems.

Red Hat Linux Python install

Red Hat is a common Linux distro used for HPC and other high reliability computing environments. Miniconda Python works well on Red Hat and many other Linux distros as well as MacOS and Windows. Thanks to .whl binary wheel packaging, many packages such as Numpy can be quickly installed via pip without compilation whether using system Python or Miniconda.

Miniconda Python install does not require “sudo”. Using EPEL or IUS, requires “sudo”. One may also load Python 3 with modules, commonly used on HPC.