Scientific Computing

Even minor point releases of Visual Studio can cause significant behavior changes and ABI breakages. The full range of defaults that changes inside Visual Studio for a major release may be more than can be accommodated with user options. Switching the Visual Studio “platform toolset” version may be of help: in Visual Studio, click Project → Properties → Platform Toolset. If the desired toolset is not present, use Visual Studio Installer to obtain it.

The standards-enforcing /permissive- flag can be turned off. That may not be enough to compile projects from older MSVC versions. The /permissive- flag is under Project → Properties → C/C++ →; Language → conformance mode.

Offline (non-Internet-connected) systems may experience failures on installing Python packages. The package authors may not have CI setup for an offline test case, so they don’t realize it’s an issue. In general, Python packages can use pyproject.toml more effectively in a Python-standard way to overcome these issues.

Instead of telling users to manually install a package such as Numpy, use pyproject.toml instead of setuptools setup_requires. Setuptools assumes the computer will be internet-connected and even if the package is already installed the install will fail on an offline system.

To ensure a package like Numpy is installed first, for example where f2py is used, have a pyproject.toml file including:

[build-system]
requires = ["setuptools>=61.0.0", "wheel", "numpy"]
build-backend = "setuptools.build_meta"

This will auto-install the prereqs before the install begins. When including this pyproject.toml parameter, do not omit “setuptools”, “wheel” or the package may fail to install.

The object-oriented Matplotlib API is slightly more verbose, but more robust than the state-machine API.

import matplotlib.pyplot as plt

f1 = plt.figure(layout='constrained')
a1 = f1.gca()
p1 = a1.plot(x,y)

a1.set_title('fun plot')
a1.set_xlabel('x [in]')
a1.set_ylabel('y [out]')

plt.show()

The OO interface avoids updating the wrong plot vs. the state machine interface where the plot in focus is updated.

import matplotlib.pyplot as plt

plt.figure()
plt.plot(x,y)
plt.title('title for figure')
plt.xlabel('x [in]')

plt.show()

“Effective Matplotlib” reference guide for moderately advanced Matplotlib graphs.

Related: datetime in Matplotlib

Pip is a widely-used, complex Python package installer program with a lot of legacy baggage. The Python Software Foundation recognizes the critical need to update Pip, putting $116K to sponsor a senior dev to modernize Pip.

Sometimes, pip install fails to realize a .whl binary wheel is available. Thus pip tries to download and install a package from source code. In the case of a large package like SciPy, OpenCV or Pillow on an embedded system like the Raspberry Pi Zero, it could take hours or even days to compile, probably failing numerous times due to missing prerequisite binary libraries.

A workaround to Pip not automatically finding the desired .whl binary wheel is to download and install the .whl directly. The binary wheels are often available at PyPI from the package download page, for example SciPy. For embedded systems such as Raspberry Pi, there may be non-PyPI sites such as PiWheels.

Download the file, then pip install from the file like:

https://www.piwheels.org/simple/scipy/scipy-1.3.2-cp37-cp37m-linux_armv7l.whl

If the wheel binary is not compatible with the system, it will fail to import or run. In that case, simply pip uninstall my_package_name and try something else.

Advanced FreeRDP options greatly improve RDP Windows Remote Desktop connections over weak internet connections.

A Linux laptop can connect to Windows PCs via SSH port forwarding using:

#!/bin/sh

ssh -f -L 4389:localhost:3389 remoteusername@remoteIP sleep 1;

xfreerdp /cert-ignore /v:localhost:4389 \
/bpp:8 /network:modem /compression -themes -wallpaper \
/clipboard /audio-mode:1 \
/auto-reconnect -glyph-cache

/clipboard

enable bidirectional clipboard

/bpp:8

uses 256 colors-low the quality, but really makes a speed improvement. Might not display videos (e.g. VLC)–try /bpp:16 if trouble.

/bpp:16

uses 65536 colors, saving bandwidth over 24-bit color with negligible visible difference for most basic uses.

/network:modem /compression

reduce bandwidth via compression (trade CPU usages for network bandwidth)

-themes -wallpaper

great speedup by not needlessly sending background graphics repeatedly

/async-update /async-input

disable RDP waiting for screen updates to reach you before it accepts input. These allow clicking ahead before the screen updates. Be careful of clicking unwanted options while using the PC.

-glyph-cache

disable glyph caching. Note: this can cause garbled characters and radio boxes.

/audio-mode:1

disable FreeRDP audio redirection (do not play sound from remote PC)

/auto-reconnect

automatically reconnect on failure (also works over SSH tunnel)

Notes

For remotely operated Digital amateur radio modes, it’s important to keep the audio generation/reception on the remote PC.

Related: RDP over SSH port forwarding

Usually it’s desired to check Python type annotations for an entire project. However, when introducing type checking to a large Python project, it can be beneficial to introduce type checking gradually, to avoid massive refactoring all at once. You will almost certainly find old code bugs due to type checking that will require refactoring, which introduces risk. Mitigate this risk by introducing type checking gradually.

MyPy is a strongly recommended type checker, though others exist. Use pyproject.toml to configure MyPy default behavior. To tell MyPy to only check certain files, use the MyPy –follow-imports= option like:

mypy --follow-imports=skip myproj/foo.py myproj/bar.py

Only certain directories can be checked like:

mypy --follow-imports=skip myproj/sub/ myproj/sub2

Once the type checks pass via mypy --follow-imports=skip, we recommend trying

mypy --follow-imports=silent

to improve robustness of the type check for those files / directories.

Enhanced mypy usage

32-bit Python binaries can run on 64-bit OS. To precisely detect if the operating system is 64-bit or 32-bit, check if Python itself is 32-bit to avoid falsely detecting a 64-bit OS as 32 bit. This check is just for the Python interpreter. To detect OS parameters in a cross-platform-robust way further checks are necessary.

python -c "import sys; print(sys.maxsize > 2**32)"

• 32-bit Python will print False
• 64-bit Python will print True

In a script:

import sys

def is_64bit() -> bool:
    return sys.maxsize > 2**32

If the history of a Git repository has been rewritten, but someone has forked the repository before the history was rewritten, they will be unable to merge due to the unrelated history. This method corrects the history in their forked branches to match the original repository.

In this synthetic example:

• Original Git repository: github.com/username/coolprog main branch develop
• Forked Git repository: github.com/jane/coolprog
• A colleague, Jane, with GitHub username jane, has created a branch from develop named feature1. However, scivision changed the history of develop after the fork.

The “easy” way can reduce the risk of compromising months or years of work compared to the “Pure Git” way.

Easy way

If the “Pure Git” way was attempted first, avoid using that directory. Instead, git clone the forked repository again.

Clone the forked repository into a temporary directory:

cd ${TMPDIR}
git clone https://github.invalid/jane/coolprog

cd coolprog
git switch feature1

Create a branch jane-feature1 in the original repository to copy the forked changes:

cd ~/code/coolprog

git switch -c jane-feature1

Manually merge the changes made in the feature1 branch using compare folders.

Note that the repository will likely have many files that are not in the fork due to .gitignore.

For any new files that need to be tracked by Git: git add filename

When satisfied with the manual merge, commit the changes in jane-feature1 to give proper credit to jane:

GIT_COMMITTER_NAME="Jane" GIT_COMMITTER_EMAIL="jane@users.noreply.github.com" git commit --author="Jane <jane@users.noreply.github.com>"

Note that Jane’s real-life email is not used to avoid exposing it to spammers. Verify the commit with git log.

Upload the changes to the Git repository:

git push -u origin jane-feature1

Merge changes into main branch

To incorporate these new features into the develop branch:

git switch develop

git merge jane-feature1

git push

Jane should create a new fork of the repository, deleting the old fork with the incorrect history to avoid repeating this process for future changes.

Reforking current repository

Jane should refork from the original GitHub repository with the corrected history to clean up the “unrelated changes” issue.

To do this, assuming no further changes have been made:

1. Delete the coolprog directory on the local machine.
2. “Delete repository” at the bottom of https://github.invalid/jane/coolprog/settings.
3. Fork the original repository again from https://github.invalid/scivision/coolprog#fork-destination-box.

This procedure is typically unnecessary and is only required when Git history has been rewritten before the fork.

Pure Git

This method requires Jane to force push and execute these commands (or grant write access to the forked repository). It is risky, so the “Easy” way is recommended.

Clone the forked repository into a temporary directory:

cd ${TMPDIR}
git clone https://github.invalid/jane/coolprog

cd coolprog
git switch develop

Add the original repository as upstream of the fork:

git remote add upstream https://github.invalid/scivision/coolprog
git fetch upstream

git rebase upstream/develop

git switch feature1
git rebase develop

If these changes are successful and the Git history is confirmed to be correct, a git push -f can be performed. Ensure secure, independent copies of the GitHub repository are available, as force-pushing overwrites Git history and may erase work.

Sometimes for Git repos with a long, complicated history this doesn’t work, and would be time consuming to fix. In that case, let’s try the “easy” way.

Matlab sparse matrices are among the classes that Matlab cannot pass to Python. The workaround requires enough RAM to hold the full matrix to pass to and from Python.

1. convert Matlab sparse to full
2. process sparse data in Python
3. convert Python sparse to full

All commands are issued in Matlab.

A = sparse(eye(5));  % test data

As = py.scipy.sparse.csc_matrix(full(A))

results in:

As =
  Python csc_matrix with properties:

                   dtype: [1×1 py.numpy.dtype]
    has_canonical_format: 1
      has_sorted_indices: 1
                     nnz: [1×1 py.int]
                   shape: [1×2 py.tuple]
                maxprint: [1×1 py.int]
                  indptr: [1×1 py.numpy.ndarray]
                 indices: [1×1 py.numpy.ndarray]
                    data: [1×1 py.numpy.ndarray]

      (0, 0)	1.0
      (1, 1)	1.0
      (2, 2)	1.0
      (3, 3)	1.0
      (4, 4)	1.0

PyGame is trivial to install on most devices. PyGame allows playing sound from Numpy arrays.

pip install pygame

pip prefers .whl binary wheel instead of compiling from source when a wheel is available.

Source compile pip (optional)

This is not usually needed

apt install git ffmpeg libsdl-image1.2-dev libsdl-mixer1.2-dev libsdl-ttf2.0-dev libsdl1.2-dev libfreetype6-dev libavformat-dev libportmidi-dev libswscale-dev

git clone https://github.com/pygame/pygame/

pip install -e pygame

Notes

• Most packages are available from pip as wheel .whl.
• Pygame Install reference