MyPy supports PEP 585 and 604, bringing concise Python 3.10 type annotation syntax to earlier Python versions. The new type annotation syntax works all supported Python versions, if each file using them has at the top:
from __future__ import annotationsSeparately, Numpy 1.20 made the long-awaited Numpy type hinting a reality.
The benefits of Python static type checking and examples have been discussed at length and widely adopted and funded by major tech companies, especially Dropbox. Python static type checking enhances code quality now and in the future by defining (constraining) variables and functions (methods).
Type enforcement can be done with assert.
Type hinting is more concise, flexible and readable than assert, with significantly less performance impact.
Type hinting is being continually enhanced in CPython, numerous IDEs and type annotation checkers.
With type hinting, the hint is right at the variable name (e.g. in the function declaration), while assert must occur in the code body.
MyPy is installed and upgraded by:
pip install -U mypyMyPy static type checker considers the following to be interchangeable (valid) due to duck typing:
int ↔ floatfloat ↔ complexNote that str is not equivalent to bytes.
Add to pyproject.toml:
[tool.mypy]
files = ["src"]assuming Python package files are under “src/” Then issue command:
python -m mypyNote this command checks the package and not the top-level scripts, which must be manually specified. Configure pyproject.toml to eliminate nuisance errors or otherwise configure mypy.
It takes a little practice to understand the messages.
Where multiple types are accepted, for example, str and pathlib.Path use typing.Union.
See the examples below.
Many times a function argument can handle more than one type. This is handled as follows:
from __future__ import annotations
from pathlib import Path
def reader(fn: Path | str) -> str:
fn = Path(fn).expanduser()
txt = fn.read_text()
return txtAnother case is where lists or tuples are used, the types within can be checked (optionally):
from __future__ import annotations
def reader(fn: Path | str) -> tuple[float, float]:
fn = Path(fn).expanduser()
txt: list[str] = fn.read_text().split(',')
latlon = (float(txt[0]), float(txt[1]))
return latlonOr perhaps dictionaries, where optionally types within can be checked:
from __future__ import annotations
def reader(fn: Path | str) -> dict[str, float]:
fn = Path(fn).expanduser()
txt: list[str] = fn.read_text().split(',')
params = {'lat': float(txt[0]),
'lon': float(txt[1])}
return paramsIf many value types are in the dictionary, or possibly some types are not yet supported for type hinting, simply use typing.Any e.g.
dict[str, typing.Any]The default where no type is declared is typing.Any, which basically means “don’t check this variable at this location in the code”.
As in C++, Python can type hint that a function must not return.
def hello() -> typing.NoReturn:
print("hello")error: Implicit return in function which does not returnThis is used for functions that always raise an error or always exit, and in general to help ensure control flow is not returned.
Unison file synchronizer works like a mashup of Dropbox, rsync and Git, although internally working via its own means. Unison attempts to remotely sync changes in files between a local and (typically) remote location. We have used Unison to manage systems on isolated remote networks. That is, where the remote network is isolated from directly accessing the Internet and so Dropbox etc. aren’t available.
Assuming one trusts Unison, this could be useful for cases where it is desirable to keep files out of the “cloud”. Unison file synchronizer has executable downloads for popular operating systems.
Red Hat is a common Linux distro used for HPC and other high reliability computing environments.
Miniconda
Python works well on Red Hat and many other Linux distros as well as macOS and Windows.
Thanks to .whl binary wheel packaging, many packages such as Numpy can be quickly installed via pip without compilation whether using system Python or Miniconda.
Miniconda Python install does not require “sudo”. Using EPEL or IUS, requires “sudo”. One may also load Python 3 with modules, commonly used on HPC.
Google’s Search Console is useful for uncovering lagging performance in web pages, including for blogs. I tend to write terse posts that address very specific issues. Often, these pages perform well, but I saw a few percent of pages being marked in Search Console as status “crawled - currently not indexed”. The underlying theme on these pages was they had too little ordinary paragraph text. If there are too many lists, headers, or preformatted text relative to plain paragraphs, this “not indexed” status is likely to be applied.
A few of these type of pages also suffered from “soft 404” status. I found these were very short pages that contained text with “error” or “missing”. I reworded those articles to avoid those terms. I made sure the titles didn’t include those terms. I also ensured there were not too many header tags relative to the text–perhaps one header at most per “page” of text.
The fix to these issues is generally to include more meaningful text–be sure an article is at least one or two full paragraphs. Add context that would help a more novice user understand why you applied that solution or approach. Avoid sensational or colloquial text as the search engines are smart enough to recognize this as low quality writing. As always, maintaining good spelling and adequate grammar help the search engine better appraise the quality of your content. Also consider short (less than 50 character) but meaningful page titles.
For long-lived blogs such as this one, there is inevitably content that is no longer relevant to anyone except for historical purposes. You may not want to simply delete these posts that you took time to research and share, but realize these old posts are costing your current performance by wasting crawler budget. The approach I take is to mark these old pages with “noindex” metadata. That allows reminiscing about old technology such as Blackberry OS 10 without degrading the performance of currently relevant content. I think of it as a soft deprecation of the content.
More convenient array broadcasting was added to Matlab years ago, removing the need for bsxfun. Python Numpy has even more advanced array indexing and broadcasting features to conserve memory, speeding computations.
When translating between Matlab and Python avoid simply replacing Matlab repmat with numpy.tile that copies data in memory. It may be possible to use numpy.newaxis or numpy.broadcast_arrays for O(1) speed and memory saving.
Homebrew’s brew cleanup saves disk space by deleting old versions of packages.
From time to time, CI macOS images get out of date with Homebrew, and auto-cleanup is triggered unintentionally upon brew install during a CI run.
There is no benefit to this on CI.
Disable Homebrew auto cleanup
by setting environment variable HOMEBREW_NO_INSTALL_CLEANUP=1.
On GitHub Actions, this is accomplished near the top of the particular .github/workflows/*.yml file:
env:
HOMEBREW_NO_INSTALL_CLEANUP: 1GCC (including G++ and Gfortran) on Windows is available via several means.
We generally prefer MSYS2 for using GCC and numerous libraries with pacman package manager.
Permuting and transposing array dimensions are a common operation used in data analysis. Ideally these operations would be done with index tricks instead of copying arrays. In Matlab, these operations copy the array (slow, expensive). In Python, these operations are O(1) an index manipulation creating a new view into an array (fast).
In Matlab, transpose and permute are distinct methods. In Matlab, transpose() is strictly for 2-D matrices, while permute() is for reordering dimensions of N-D arrays.
In Python, xarray and Numpy arrays are popular.
Both use the .transpose() method for N-D array dimension reordering–there is no separate “permute” method.
However, the syntax is distinct between xarray and Numpy.
xarray.Dataset
and
xarray.DataArray
have a .transpose(*dims) method.
# A.dims == ('x', 'y', 'z')
B = A.transpose(('z', 'y', 'x'))
# this is equivalent
B = A.transpose('z', 'y', 'x')
# B.dims == ('z', 'y', 'x')Numpy transpose can also permute by specifying a tuple of the desired order. Omitting the axes order argument simply reverses the order of the axes for N-dimensional arrays.
B = A.transpose((2,1,0))
D = C.transpose()The latest CMake release may be installed by any one of:
winget install Kitware.CMakepip install cmakeThe CMake build process is the same across operating systems and compilers:
cmake -S myproj -B myproj/build
cmake --build myproj/build
cmake --install myproj/buildOn Windows, CMake defaults to Visual Studio and Nmake, which may not work on some projects. Ninja build system with CMake is generally faster to build and particularly rebuild regardless of operating system. Ninja on Windows solves numerous issues vs. GNU Make. Ninja works with Visual Studio as well.
Override the default CMake generator by setting environment variable
CMAKE_GENERATOR=NinjaCMAKE_GENERATOR can be overridden (e.g. to use GNU Make from MSYS2) like:
cmake -G "MinGW Makefiles"Older CMake on Windows may get the message below. Fix by upgrading to CMake ≥ 3.17.
sh.exe was found in your PATH
For MinGW make to work correctly sh.exe must NOT be in your path.
Run cmake from a shell that does not have sh.exe in your PATH.
If you want to use a UNIX shell, then use MSYS Makefile