Scientific Computing

Note: it may be necessary to reinstall Miniconda from scratch if packages break during a Python version upgrade. Consider this before attempting an in-place Python upgrade. There is often a couple month delay between a major Python release and Anaconda defaulting to the new version.

Use the new Python version in a new conda environment by:

conda create -n py3x python=3.x

switch to this environment by

conda activate py3x

Legacy hard-coded GUIs using external libraries have considerable overhead to maintain, and suffer bit rot far faster than the underlying code. At the very least, be sure your code is usable from the command line and/or as a plain importable module without requiring a graphics library. In stripped-down cloud / embedded applications, unused graphical imports cause stability problems and considerable startup lag.

To make a Matlabplot figure axis have integer-only label ticks, use method like:

ax.yaxis.set_major_locator(MaxNLocator(integer=True))
# or
ax.xaxis.set_major_locator(MaxNLocator(integer=True))

A complete standalone example follows:

import numpy as np
import matplotlib.pyplot as plt
from matplotlib.ticker import MaxNLocator

x = np.arange(0.1,10.5,0.1) # arbitrary data

fg = plt.figure(layout='constrained')
ax = fg.gca()
ax.plot(x)

ax.yaxis.set_major_locator(MaxNLocator(integer=True))

plt.show()

If too few ticks are displayed, as per the Matplotlib MaxNLocator, you must have “at least min_n_ticks integers…found within the view limits.” Set “MaxNLocator(nbins=5,integer=True)” or similar if the defaults aren’t forcing integer ticks.

PyTest flips on DeprecationWarning and PendingDeprecationWarning as typically those running PyTest are a developer or advanced users. When the package uses warnings.warn to emit a DeprecationWarning, it can be hard to know from where in the tested package a warning is coming from.

To turn on a large amount of warnings, similar to what might be seen on CI:

python -Walways::DeprecationWarning -m pytest

If there is only one type of DeprecationWarning being omitted, a simple way to find the source of the warning is Python warning control -W. This will raise an exception at the warning with traceback:

python -Werror::DeprecationWarning -m pytest

However, often there are multiple DeprecationWarning emitted from different sources, and chances are the one of interest isn’t the first. Perhaps the warning comes from the Python distribution (e.g. Miniconda) itself. In that case, one can insert temporary warning trap into the test function or the user function:

warnings.filterwarnings("error", category=DeprecationWarning)

This might have to be done iteratively to get past the point where the uninteresting DeprecationWarning happen until you home in on the location of the DeprecationWarning of interest.

Related: silence PyTest DeprecationWarning

Git 2.31 added the ability to maximally resolve the parts of a merge commit, where some parts could not be auto-merged. This workflow isn’t good for some types of project, but a lot of projects and devs use this setting.

Set this globally (it can also be set per-repo and/or per merge tool):

git config --global mergetool.hideResolved true

This procedure works for Matlab on most Linux systems including Ubuntu and RHEL.

Extract all files from the Matlab installer archive, and run (without sudo)

./install

Install Matlab under “/home/username/.local/” since Matlab is tied to the Linux username anyway. The user will have problems updating Matlab or installing Add-On Toolboxes if Matlab is installed outside the user home directory. Install Symbolic Links to “/home/username/.local/bin/” when asked by the GUI.

Add to the user ~/.profile:

export PATH=$PATH:$HOME/.local/bin

Optional: Add links in desktop menu

Matlab can be started from Terminal:

matlab

It’s best to run non-interactive Matlab jobs (including CI) like

matlab -batch myscript

NOTE: A GUI is required for normal install

If installing Matlab remotely over SSH, use any one of:

• X11 Forwarding
• VNC
• Silent install via config file (tricky)

If “click here” isn’t working for individual site license, try dragging the “click here” to the web browser address bar.

If getting error:

terminate called after throwing an instance of 'std::runtime_error'
  what():  Unable to launch the MATLABWindow application

try removing libcrypto.so.1.1 under the Matlab install folder “bin/glnxa64/” and ./install again.

reference

Matlab can extract .zst Zstd files, assuming the Zstd program is installed. Zstd is widely supported and available across operating systems. Use extract_zstd Matlab function to extract Zstd files.

Related: Python extract .zst file

FFmpeg can convert audio files to Opus format. Opus is playable in many web browsers and players in general. Opus is highly effective across a wide range of bitrates, from low (≪ 64kbps) for speech and high music bitrates.

ffmpeg -i in.wav -codec:a libopus out.opus

Typically it’s desired to set a specific output bitrate. The bitrate affects the audio quality–lower bitrate sounds more flat and mechanical.

ffmpeg -i in.wav -codec:a libopus  -b:a 32k out.opus

sets output bitrate to 32kbps, suitable for speech or lower quality music.

Developers often use scripting languages for common tasks and utilities outside the main program language or compiled executables. For example, Git is a popular C program that uses numerous Python scripts for utility functions. Scripting is commonly used where it’s burdensome to implement non-core functionality in the main project language. There can be a runtime speed penalty for scripts, which is why popular utilities may eventually be recoded in the compiled project language.

As in spoken and written language, there is no one scripting language that will suit all environments. Nonetheless, there are scripting language choices that more closely meet universal applicability with today’s computing environments. When thinking of long-term reproducibility, scripting is also a factor to consider.

For example, Perl, Python and Ruby are long-popular scripting languages that aren’t installed by default on Windows. Powershell is installed by default on Windows and is available on macOS and Linux, but not installed on the latter two by default. The default shell varies between operating systems and distros. For development-oriented work we use CMake scripts instead of Python for tasks that CMake is a better fit for, such as finding and executing programs.

I don’t think there will practically ever be a universal scripting language, as the language would have to be universally installed by default across operating systems and distros to be immediately useful. Instead, we can have practical scripting languages targeted toward specific communities. For example, where a project already requires CMake and the scripting task is a good fit for CMake script, it may make sense to put utility scripts in CMake instead of Python for example. This can be true even for mixed Python-CMake projects. Sometimes we default to put every script in Python, when it may be easier and make more sense to put a subset of scripts in CMake instead.

CMake scripts are executed with (optional) options like:

cmake -Dvar=value -P myscript.cmake

Like with shell scripts, standalone CMake scripts can have a first-line shebang that tells the shell what program should execute the script.

#!/usr/bin/env -S cmake -P

While there’s no harm in adding this shebang to CMake scripts, is there a way to pass dynamic “-Dvar=value” to the script when using the shebang like “./myscript.cmake”? If the script is OK without such parameters, then the shebang can be useful.

CMAKE_COMMAND can be used to directly invoke CMake scripts with options. This is particularly useful to chain CMake scripts during CTest runs with fixtures. It’s important to set the minimum CMake version in the CMake script file as otherwise CMake defaults to CMake < 2.6 policies. That makes current CMake syntax fail in puzzling ways.

Example: Suppose a test script that reads a file using regex. This will fail without cmake_minimum_required() or cmake_policy(). I prefer cmake_minimum_required to make the script fail if someone uses the script directly with too-old CMake.

add_test(NAME myRegex COMMAND ${CMAKE_COMMAND} -P ${CMAKE_CURRENT_SOURCE_DIR}/mytest.cmake)

mytest.cmake contains:

cmake_minimum_required(VERSION 3.10)

file(STRINGS example.txt m REGEX "([\.A-Za-z0-9_]+)")

The failure will be over escaped \. if cmake_minimum_version or cmake_policy isn’t used in mytest.cmake.