Note: it may be necessary to reinstall
Anaconda/Miniconda
from scratch if packages break during a Python version upgrade.
Consider this before attempting an in-place Python upgrade.
There is often a couple month delay between a major Python release and Anaconda defaulting to the new version.
Use the new Python version in a new conda environment by:
conda create -n py3x python=3.x
switch to this environment by
conda activate py3x
For interactive data science GUI applications
Jupyter
is often recommended.
Legacy hard-coded GUIs using external libraries have considerable overhead to maintain, and suffer bit rot far faster than the underlying code.
At the very least, be sure your code is usable from the command line and/or as a plain importable module without requiring a graphics library.
In stripped-down cloud / embedded applications, unused graphical imports cause stability problems and considerable startup lag.
To make a Matlabplot figure axis have integer-only labels, use method like:
ax.yaxis.set_major_locator(MaxNLocator(integer=True))
# or
ax.xaxis.set_major_locator(MaxNLocator(integer=True))
A complete standalone example follows:
import numpy as np
from matplotlib.figure import Figure
from matplotlib.ticker import MaxNLocator
x = np.arange(0.1,10.5,0.1) # arbitrary data
fg = Figure()
ax = fg.gca()
ax.plot(x)
ax.yaxis.set_major_locator(MaxNLocator(integer=True))
fg.savefig("example.png")
If too few ticks are displayed, as per the
Matplotlib MaxNLocator,
you must have “at least min_n_ticks integers…found within the view limits.”
Set “MaxNLocator(nbins=5,integer=True)” or similar if the defaults aren’t forcing integer ticks.
PyTest flips on DeprecationWarning and PendingDeprecationWarning as typically those running PyTest are a developer or advanced users.
When the package uses warnings.warn to emit a DeprecationWarning, it can be hard to know from where in the tested package a warning is coming from.
To turn on a large amount of warnings, similar to what might be seen on CI:
python -Walways::DeprecationWarning -m pytest
If there is only one type of DeprecationWarning being omitted, a simple way to find the source of the warning is
Python warning control-W.
This will raise an exception at the warning with traceback:
python -Werror::DeprecationWarning -m pytest
However, often there are multiple DeprecationWarning emitted from different sources, and chances are the one of interest isn’t the first.
Perhaps the warning comes from the Python distribution (e.g. Miniconda) itself.
In that case, one can insert temporary
warning trap
into the test function or the user function:
This might have to be done iteratively to get past the point where the uninteresting DeprecationWarning happen until you home in on the location of the DeprecationWarning of interest.
Git 2.31 added the ability to
maximally resolve
the parts of a merge commit, where some parts could not be auto-merged.
This workflow isn’t good for some types of project, but a lot of projects and devs do enjoy this setting.
Set this globally (it can also be set per-repo and/or per merge tool):
Matlab can extract .zst Zstd files, assuming the Zstd program is installed.
Zstd is widely supported and available across operating systems.
Use our extract_zstd Matlab function to extract Zstd files.
import matlab.engine
eng = matlab.engine.start_matlab('-nojvm')
y = eng.asin(1.)
eng.quit()
The Matlab Engine should take less than 1 second for Matlab Engine to start when called from Python.
Note: some Matlab functions require JVM, thus although much slower to start, you may need to remove the -nojvm option.
Many Matlab numeric classes (single, double, logical) can be converted to Python types like:
numpy.asarray(x)
Python floats pass into Matlab Engine by including a period . after the number.
asin(1) fails
asin(1.) works
Python can pass N-dimensional arrays to Matlab.
Matlab Engine provides asynchronous call with
background=True
Zstd is an open file compression
standard.
Zstd has become widely used and is incorporated in the
Linux kernel.
We use Zstd for data archiving particularly for large files where size and speed are a concern.
CMake ≥ 3.15 supports Zstd compression.
CMake 3.18 added
file(ARCHIVE_CREATE) and file(ARCHIVE_EXTRACT)
that also support Zstd.
Zstd is
vendored
into CMake, so there is no need to worry about system shared libraries for Zstd.
The current method to create “archive” of all files under a directory “in_dir” currently requires syntax like:
# need working_directory ${in} to avoid computer-specific relative paths
# use . not ${in} as last argument to avoid more relative path issues
set(archive"my.zst")set(in_dir"data/")execute_process(
COMMAND${CMAKE_COMMAND}-Etarc${archive}--zstd.WORKING_DIRECTORY${in_dir})
The syntax to create traditional .zip files is very similar:
The newer interface below is not yet usable because there’s no control over nuisance computer-specific relative directories added.
The Kitware contributor intends to fix this by adding a WORKING_DIRECTORY–check to see if this has been done yet.
set(archive"my.zst")set(in_dir"data/")file(ARCHIVE_CREATEOUTPUT${archive}PATHS${in_dir}COMPRESSIONZstdCOMPRESSION_LEVEL3)# level is arbitrary, bigger is more compressed
For the past several years, Microsoft provides legal free Windows 10 virtual machine image downloads.
The caveat is the image is only good for 90 days.
A fresh free VM image setup is needed every 90 days.
VM snapshot does NOT workaround the 90 day timer.
unzip MSEdge.Win10*.zip creates an .ova file if you picked VirtualBox.
Import image into the virtual machine software.
If using VirtualBox: check “Reinitialize the MAC address of all network cards”
Take a snapshot BEFORE first starting the VM to avoid having to repeat the previous three steps after 90 days.
You will lose all your subsequent Windows programs/setup every 90 days.
FFmpeg can
convert
audio files to
Opus
format.
Opus is playable in many web browsers and players in general.
Opus is highly effective across a wide range of bitrates, from low (≪ 64kbps) for speech and high music bitrates.
ffmpeg -i in.wav -codec:a libopus out.opus
Typically it’s desired to set a specific output bitrate.
The bitrate affects the audio quality–lower bitrate sounds more flat and mechanical.