Upgrade Anaconda for latest Python

Note: it may be necessary to reinstall Anaconda/Miniconda from scratch if packages break during a Python version upgrade. Consider this before attempting an in-place Python upgrade. There is often a couple month delay between a major Python release and Anaconda defaulting to the new version.

Use the new Python version in a new conda environment by:

conda create -n py3x python=3.x

switch to this environment by

conda activate py3x

For interactive data science GUI applications Jupyter is often recommended. Legacy hard-coded GUIs using external libraries have considerable overhead to maintain, and suffer bit rot far faster than the underlying code. At the very least, be sure your code is usable from the command line and/or as a plain importable module without requiring a graphics library. In stripped-down cloud / embedded applications, unused graphical imports cause stability problems and considerable startup lag.

Related: Why upgrade Python?

Force integer axis labels on Matplotlib

To make a Matlabplot figure axis have integer-only labels, use method like:

ax.yaxis.set_major_locator(MaxNLocator(integer=True))
# or
ax.xaxis.set_major_locator(MaxNLocator(integer=True))

A complete standalone example follows:

import numpy as np
from matplotlib.figure import Figure
from matplotlib.ticker import MaxNLocator

x = np.arange(0.1,10.5,0.1) # arbitrary data

fg = Figure()
ax = fg.gca()
ax.plot(x)

ax.yaxis.set_major_locator(MaxNLocator(integer=True))

fg.savefig("example.png")

If too few ticks are displayed, as per the Matplotlib MaxNLocator, you must have “at least min_n_ticks integers…found within the view limits.” Set “MaxNLocator(nbins=5,integer=True)” or similar if the defaults aren’t forcing integer ticks.

Find source of PyTest warning

PyTest flips on DeprecationWarning and PendingDeprecationWarning as typically those running PyTest are a developer or advanced users. When the package uses warnings.warn to emit a DeprecationWarning, it can be hard to know from where in the tested package a warning is coming from.

To turn on a large amount of warnings, similar to what might be seen on CI:

python -Walways::DeprecationWarning -m pytest

If there is only one type of DeprecationWarning being omitted, a simple way to find the source of the warning is Python warning control -W. This will raise an exception at the warning with traceback:

python -Werror::DeprecationWarning -m pytest

However, often there are multiple DeprecationWarning emitted from different sources, and chances are the one of interest isn’t the first. Perhaps the warning comes from the Python distribution (e.g. Miniconda) itself. In that case, one can insert temporary warning trap into the test function or the user function:

warnings.filterwarnings("error", category=DeprecationWarning)

This might have to be done iteratively to get past the point where the uninteresting DeprecationWarning happen until you home in on the location of the DeprecationWarning of interest.


Related: silence PyTest DeprecationWarning

Git 2.31 mergetool enhancements

Git 2.31 added the ability to maximally resolve the parts of a merge commit, where some parts could not be auto-merged. This workflow isn’t good for some types of project, but a lot of projects and devs do enjoy this setting.

Set this globally (it can also be set per-repo and/or per merge tool):

git config --global mergetool.hideResolved true

Install Matlab in Linux

This procedure works for Matlab (regular and student) on most Linux systems.

Extract all files from the Matlab installer archive, and run (without sudo)

./install

Install under “~/.local/” since it’s tied to your Linux username anyway. Install Symbolic Links to “~/.local/bin/” when asked by the GUI.

Add to ~/.bashrc:

export PATH=$PATH:$HOME/.local/bin

Optional: Add links in your desktop menu


Matlab can be started from Terminal:

matlab

It’s best to run non-interactive Matlab jobs (including CI) like

matlab -batch myscript

NOTE: A GUI is required for normal install

If installing Matlab remotely over SSH, use any one of:


If you cannot click the “click here” for individual site license, try dragging the “click here” to the web browser address bar.


If getting error:

terminate called after throwing an instance of 'std::runtime_error'
  what():  Unable to launch the MATLABWindow application

try removing file “bin/glnxa64/libcrypto.so.1.1” and ./install again.

reference

Install Matlab Engine API in Python

Matlab Engine API allows calling Matlab functions from Python code.

Find the directory Matlab is installed in. From Terminal / PowerShell:

matlab -batch matlabroot

change to the directory indicated then

cd extern/engines/python

Execute these commands (not in Matlab prompt):

  • MacOS or Linux:

    python setup.py build --build-base=$(mktemp -d) install --user
    
  • Windows Powershell:

    python setup.py build --build-base=$env:temp install --user
    

Root / admin permissions are NOT needed.


A simple example of using Matlab from Python:

import matlab.engine
eng = matlab.engine.start_matlab('-nojvm')
y = eng.asin(1.)

eng.quit()

The Matlab Engine should take less than 1 second for Matlab Engine to start when called from Python. Note: some Matlab functions require JVM, thus although much slower to start, you may need to remove the -nojvm option.

Many Matlab numeric classes (single, double, logical) can be converted to Python types like:

numpy.asarray(x)

Python floats pass into Matlab Engine by including a period . after the number.

  • asin(1) fails
  • asin(1.) works

Python can pass N-dimensional arrays to Matlab.

Matlab Engine provides asynchronous call with background=True


References:

Related:

CMake Zstd compression

Zstd is an open file compression standard. Zstd has become widely used and is incorporated in the Linux kernel. We use Zstd for data archiving particularly for large files where size and speed are a concern. CMake ≥ 3.15 supports Zstd compression. CMake 3.18 added file(ARCHIVE_CREATE) and file(ARCHIVE_EXTRACT) that also support Zstd. Zstd is vendored into CMake, so there is no need to worry about system shared libraries for Zstd.

The current method to create “archive” of all files under a directory “in_dir” currently requires syntax like:

# need working_directory ${in} to avoid computer-specific relative paths
# use . not ${in} as last argument to avoid more relative path issues

set(archive "my.zst")
set(in_dir "data/")

execute_process(
  COMMAND ${CMAKE_COMMAND} -E tar c ${archive} --zstd .
  WORKING_DIRECTORY ${in_dir})

The syntax to create traditional .zip files is very similar:

set(archive "my.zip")

execute_process(
    COMMAND ${CMAKE_COMMAND} -E tar c ${archive} --format=zip .
    WORKING_DIRECTORY ${in_dir})

The newer interface below is not yet usable because there’s no control over nuisance computer-specific relative directories added. The Kitware contributor intends to fix this by adding a WORKING_DIRECTORY–check to see if this has been done yet.

set(archive "my.zst")
set(in_dir "data/")

file(ARCHIVE_CREATE
  OUTPUT ${archive}
  PATHS ${in_dir}
  COMPRESSION Zstd
  COMPRESSION_LEVEL 3)
# level is arbitrary, bigger is more compressed

NOTE: The “FORMAT” option is not used for Zstd.

Free Windows 10 Virtual Machine images

For the past several years, Microsoft provides legal free Windows 10 virtual machine image downloads. The caveat is the image is only good for 90 days. A fresh free VM image setup is needed every 90 days. VM snapshot does NOT workaround the 90 day timer.

unzip MSEdge.Win10*.zip creates an .ova file if you picked VirtualBox. Import image into the virtual machine software. If using VirtualBox: check “Reinitialize the MAC address of all network cards” Take a snapshot BEFORE first starting the VM to avoid having to repeat the previous three steps after 90 days. You will lose all your subsequent Windows programs/setup every 90 days.


Alternative: ReactOS virtual machines

FFmpeg Opus .ogg output

FFmpeg can convert audio files to Opus format. Opus is playable in many web browsers and players in general. Opus is highly effective across a wide range of bitrates, from low (≪ 64kbps) for speech and high music bitrates.

ffmpeg -i in.wav -codec:a libopus out.opus

Typically it’s desired to set a specific output bitrate. The bitrate affects the audio quality–lower bitrate sounds more flat and mechanical.

ffmpeg -i in.wav -codec:a libopus  -b:a 32k out.opus

sets output bitrate to 32kbps, suitable for speech or lower quality music.