Scientific Computing

Use PyPI to distribute Python packages

This procedure is for PyPI Warehouse.

Assuming a Python package named “myprogram”, have a directory structure like

pyproject.toml
myprogram/
   __init__.py

A minimal pyproject.toml is all that’s required for a PyPI Python package. The package “version” number must be distinct for each release, or PyPI will prevent upload.

Ensure prereqs are installed:

pip install --upgrade twine build

One-time setup: sign up for a PyPI account. Create ~/.pypirc with the PyPI token.

In the project directory, create the package:

python -m build

Upload the package to PyPI:

python -m twine upload dist/*

Now the package is live to the world on PyPI, installable via pip install.

HDF5 on Intel oneAPI for Windows

Intel oneAPI on Windows provides an easy way to use Fortran MPI on Windows. The Intel oneAPI compile and link commands on Windows are distinct from those on Linux, perhaps reflecting the internal use of Visual Studio on Windows. The HDF5 1.10.6 release changed the naming convention for the HDF5 Fortran library files on all operating systems.

  • old: hdf5hl_fortran.
  • new: hdf5_hl_fortran.

If experiencing trouble finding HDF5 with CMake, try FindHDF5.cmake. FindHDF5 specifically works with Intel compilers and HDF5 across operating systems including Windows.

To build HDF5 library, including on Windows, use CMake to build HDF5.


If you use include/static you will get errors like

error LNK2019: unresolved external symbol H5GLOBAL_mp_H5F_ACC_TRUNC_F referenced in function MAIN__

ConnectBot cannot import OpenSSH keys

The free open source SSH app ConnectBot allows connecting to SSH servers with port forwarding using public key authentication, including ED25519.

Sometimes it’s necessary to share SSH keypairs on multiple clients. Perhaps the server owner isn’t willing to bother with more than one SSH client key. ConnectBot has an open issue since it cannot import OpenSSH keys created on a PC.

Generally users should create unique SSH public/private keypairs for each device. Sharing keys between devices means if a device is compromised, deleting its key from ~/.ssh/authorized_keys on the SSH server disables all other devices sharing that key.

Workaround by creating an SSH keypair in ConnectBot. Copy the ConnectBot-created public/private keypair to the PC ~/.ssh directory.

The stem (filename without extension) of the public and private keys must match. The public key should have a .pub suffix, while the private key has no suffix.

Thereby the same SSH keypair is used on the phone with ConnectBot and the PC with OpenSSH client.

git diff feature branch with unstaged changes

To integrate changes from branch “feature1” without cherry picking can be done file by file or line by line.

Compare current unstaged changes file-by-file, with the ability to line-by-line implement changes:

git difftool feature1

Whole branch compare does NOT allow line-by-line edits, so perhaps open a second terminal and “git checkout” files from the feature branch as desired.

git difftool -d feature1

Meld and Visual Studio Code are each good GUIs for Git difftool and mergetool.

USA 2G cellular shutdown

Carriers in numerous countries worldwide have shutdown 2G networks to free spectrum for 4G and 5G services. Embedded modems such as automotive (OnStar) and alarm systems may again be impacted by these shutdowns. In developing regions 2G networks may linger for several more years, due to cost-effective legacy devices.

Those designing IoT and other embedded devices with cellular modems should consider LPWA 4G LTE, particularly Cat M1 and NB1 to help ensure global functionality. Each geopolitical region has unique LTE bands, but often OEM modules with the same pinout have region-specific models. OEM LTE modules will incorporate at least some bands for each region, so that global LTE roaming for even inexpensive LTE modems will become increasingly common.

2G shutdown dates from Digi

Notes

The USA 1G AMPS and 2G D-AMPS networks were shutdown in 2008. In June 2013, Sprint shutdown the Nextel iDEN network for LTE band 26.

Recovering from broken Git repo

A local Git repo can become corrupted in rare circumstances, perhaps doing a git commit just as a computer crashes or loses power. A common symptom of a corrupted local Git repo is any Git command except for perhaps git diff giving error:

fatal: your current branch appears to be broken

The changes are likely still present, as seen via git diff. Previous commits that were not pushed to remote are likely present as well in the form of the modified files, but the historical local Git commit deltas may not be recoverable. This recovery will in effect “squash” the local commits that weren’t previously pushed to remote.

NOTE: work done on other branches that weren’t pushed to remote may not be retrievable.

Recovery

  1. Copy the directory tree of the affected local Git repo, preferably on another hard drive or in the cloud.
  2. git clone a fresh copy of the remote Git repo to a new directory
  3. Compare folders to incrementally copy into the new directory the changes from the old corrupted directory. If there are a large number of changes, consider making the changes via multiple Git commits.

Notes

Add missing LaTeX fonts

Missing LaTeX fonts can be added via TeXLive or MikTeX. Using the symbolic fonts is as easy as:

\documentclass[a4paper,12pt]{article}

\usepackage{fontawesome5}

\begin{document}
  \faGithub This is a GitHub logo.
\end{document}

This may need to use XeLaTeX. We use fontawesome5 instead of obsolete fontawesome, which is version 4. FontAwesome5 is in TeXLive 2018.

TeXLive is popular across operating systems. Linux users can use system package managers to install groups of TeXLive packages. Advanced Linux TeXLive users can independently setup TeXLive to get the latest packages individually, to save hundreds of megabytes of install space.


For marvosym:

Windows/Linux tlmgr:

tlmgr install marvosym

Linux: apt:

apt install texlive-fonts-recommended

For fontawesome5:

FontAwesome gives popular emoji and website icons.

Windows/Linux: tlmgr:

tlmgr install fontawesome5

Linux: apt:

apt install texlive-fonts-extra

MikTeX is another LaTeX distro on Windows. MiKTeX will automatically install packages as needed. If that isn’t happening, check:

Start → MiKTeX Console → Settings → General → “Always install missing packages on the fly”

Otherwise, manually search for package name and install:

Start → MiKTeX Console → Packages

Install OpenCV in Python

The unofficial OpenCV PyPI wheels work with pip install methods:

pip install opencv-python

For ARM / Raspberry Pi:

pip install opencv-python

also works for certain ARM platforms like the Raspberry Pi.

OpenCV is trivial and fast to install on a Raspberry Pi via pip as described above.

For the latest extended functionality that hasn’t yet been incorporated into the core package, OpenCV including the Extra contributed modules may be obtained by:

pip install opencv-contrib-python

If ... is not supported on this platform error be sure it’s using desired Python install. One may have to manually specify the path for the pip command e.g.

$Env:SystemDrive\anaconda3\scripts\python -m pip install opencv-python

We made several test scripts to try out the OpenCV install. Compiling OpenCV yourself allows customizing and optimizing OpenCV for your computer (e.g. using GPU, TBB, OpenCL, etc.).

The conda install opencv and conda install -c conda-forge opencv methods for OpenCV continue to be BROKEN for video/image reading and display. Use pip install above instead.

Convert Periscope video for YouTube

If YouTube won’t accept a video upload, or the video never completes “Processing” on YouTube, try re-encoding the video with FFmpeg. Sometimes lossy conversion is necessary to achieve the YouTube recommended upload settings.

Periscope video downloads use MPEG TS container with .ts file extension. These .ts files can be played back in VLC or similar to confirm content. A lossless conversion to YouTube is possible with:

ffmpeg -i pscp.ts -bsf:a aac_adtstoasc -codec copy -max_muxing_queue_size 1000 pscp.mp4

The FFmpeg option -max_muxing_queue_size is an arbitrary parameter. The default queue size is sometimes not large enough:

Too many packets buffered for output stream 0:1.
  • Do not use .mkv extension for YouTube uploads, it will fail to process.
  • certain formats like AV1 or FFV1 will fail to finish processing the upload
  • after completing Uploading, video should start Processing in less than 5 minutes. If not, it will probably never actually work.

Uploads that don’t finish conversion processing after upload are not viewable on YouTube.