Scientific Computing

The object-oriented Matplotlib API is slightly more verbose, but more robust than the state-machine API.

import matplotlib.pyplot as plt

f1 = plt.figure(layout='constrained')
a1 = f1.gca()
p1 = a1.plot(x,y)

a1.set_title('fun plot')
a1.set_xlabel('x [in]')
a1.set_ylabel('y [out]')

plt.show()

The OO interface avoids updating the wrong plot vs. the state machine interface where the plot in focus is updated.

import matplotlib.pyplot as plt

plt.figure()
plt.plot(x,y)
plt.title('title for figure')
plt.xlabel('x [in]')

plt.show()

“Effective Matplotlib” reference guide for moderately advanced Matplotlib graphs.

Related: datetime in Matplotlib

Pip is a widely-used, complex Python package installer program with a lot of legacy baggage. The Python Software Foundation recognizes the critical need to update Pip, putting $116K to sponsor a senior dev to modernize Pip.

Sometimes, pip install fails to realize a .whl binary wheel is available. Thus pip tries to download and install a package from source code. In the case of a large package like SciPy, OpenCV or Pillow on an embedded system like the Raspberry Pi Zero, it could take hours or even days to compile, probably failing numerous times due to missing prerequisite binary libraries.

A workaround to Pip not automatically finding the desired .whl binary wheel is to download and install the .whl directly. The binary wheels are often available at PyPI from the package download page, for example SciPy. For embedded systems such as Raspberry Pi, there may be non-PyPI sites such as PiWheels.

Download the file, then pip install from the file like:

https://www.piwheels.org/simple/scipy/scipy-1.3.2-cp37-cp37m-linux_armv7l.whl

If the wheel binary is not compatible with the system, it will fail to import or run. In that case, simply pip uninstall my_package_name and try something else.

Advanced FreeRDP options greatly improve RDP Windows Remote Desktop connections over weak internet connections.

A Linux laptop can connect to Windows PCs via SSH port forwarding using:

#!/bin/sh

ssh -f -L 4389:localhost:3389 remoteusername@remoteIP sleep 1;

xfreerdp /cert-ignore /v:localhost:4389 \
/bpp:8 /network:modem /compression -themes -wallpaper \
/clipboard /audio-mode:1 \
/auto-reconnect -glyph-cache

/clipboard

enable bidirectional clipboard

/bpp:8

uses 256 colors-low the quality, but really makes a speed improvement. Might not display videos (e.g. VLC)–try /bpp:16 if trouble.

/bpp:16

uses 65536 colors, saving bandwidth over 24-bit color with negligible visible difference for most basic uses.

/network:modem /compression

reduce bandwidth via compression (trade CPU usages for network bandwidth)

-themes -wallpaper

great speedup by not needlessly sending background graphics repeatedly

/async-update /async-input

disable RDP waiting for screen updates to reach you before it accepts input. These allow clicking ahead before the screen updates. Be careful of clicking unwanted options while using the PC.

-glyph-cache

disable glyph caching. Note: this can cause garbled characters and radio boxes.

/audio-mode:1

disable FreeRDP audio redirection (do not play sound from remote PC)

/auto-reconnect

automatically reconnect on failure (also works over SSH tunnel)

Notes

For remotely operated Digital amateur radio modes, it’s important to keep the audio generation/reception on the remote PC.

Related: RDP over SSH port forwarding

Usually it’s desired to check Python type annotations for an entire project. However, when introducing type checking to a large Python project, it can be beneficial to introduce type checking gradually, to avoid massive refactoring all at once. You will almost certainly find old code bugs due to type checking that will require refactoring, which introduces risk. Mitigate this risk by introducing type checking gradually.

MyPy is a strongly recommended type checker, though others exist. Use pyproject.toml to configure MyPy default behavior. To tell MyPy to only check certain files, use the MyPy –follow-imports= option like:

mypy --follow-imports=skip myproj/foo.py myproj/bar.py

Only certain directories can be checked like:

mypy --follow-imports=skip myproj/sub/ myproj/sub2

Once the type checks pass via mypy --follow-imports=skip, we recommend trying

mypy --follow-imports=silent

to improve robustness of the type check for those files / directories.

Enhanced mypy usage

32-bit Python binaries can run on 64-bit OS. To precisely detect if the operating system is 64-bit or 32-bit, check if Python itself is 32-bit to avoid falsely detecting a 64-bit OS as 32-bit. This check is just for the Python interpreter. To detect OS parameters in a cross-platform-robust way further checks are necessary.

python -c "import sys; print(sys.maxsize > 2**32)"

• 32-bit Python will print False
• 64-bit Python will print True

In a script:

import sys

def is_64bit() -> bool:
    return sys.maxsize > 2**32

If the history of a Git repository has been rewritten, but someone has forked the repository before the history was rewritten, they will be unable to merge due to the unrelated history. This method corrects the history in their forked branches to match the original repository.

In this synthetic example:

• Original Git repository: github.com/username/coolprog main branch develop
• Forked Git repository: github.com/jane/coolprog
• A colleague, Jane, with GitHub username jane, has created a branch from develop named feature1. However, scivision changed the history of develop after the fork.

The “easy” way can reduce the risk of compromising months or years of work compared to the “Pure Git” way.

Easy way

If the “Pure Git” way was attempted first, avoid using that directory. Instead, git clone the forked repository again.

Clone the forked repository into a temporary directory:

cd ${TMPDIR}
git clone https://github.invalid/jane/coolprog

cd coolprog
git switch feature1

Create a branch jane-feature1 in the original repository to copy the forked changes:

cd ~/code/coolprog

git switch -c jane-feature1

Manually merge the changes made in the feature1 branch using compare folders.

Note that the repository will likely have many files that are not in the fork due to .gitignore.

For any new files that need to be tracked by Git: git add filename

When satisfied with the manual merge, commit the changes in jane-feature1 to give proper credit to jane:

GIT_COMMITTER_NAME="Jane" GIT_COMMITTER_EMAIL="jane@users.noreply.github.com" git commit --author="Jane <jane@users.noreply.github.com>"

Note that Jane’s real-life email is not used to avoid exposing it to spammers. Verify the commit with git log.

Upload the changes to the Git repository:

git push -u origin jane-feature1

Merge changes into main branch

To incorporate these new features into the develop branch:

git switch develop

git merge jane-feature1

git push

Jane should create a new fork of the repository, deleting the old fork with the incorrect history to avoid repeating this process for future changes.

Reforking current repository

Jane should refork from the original GitHub repository with the corrected history to clean up the “unrelated changes” issue.

To do this, assuming no further changes have been made:

1. Delete the coolprog directory on the local machine.
2. “Delete repository” at the bottom of https://github.invalid/jane/coolprog/settings.
3. Fork the original repository again from https://github.invalid/scivision/coolprog#fork-destination-box.

This procedure is typically unnecessary and is only required when Git history has been rewritten before the fork.

Pure Git

This method requires Jane to force push and execute these commands (or grant write access to the forked repository). It is risky, so the “Easy” way is recommended.

Clone the forked repository into a temporary directory:

cd ${TMPDIR}
git clone https://github.invalid/jane/coolprog

cd coolprog
git switch develop

Add the original repository as upstream of the fork:

git remote add upstream https://github.invalid/scivision/coolprog
git fetch upstream

git rebase upstream/develop

git switch feature1
git rebase develop

If these changes are successful and the Git history is confirmed to be correct, a git push -f can be performed. Ensure secure, independent copies of the GitHub repository are available, as force-pushing overwrites Git history and may erase work.

Sometimes for Git repos with a long, complicated history this doesn’t work, and would be time consuming to fix. In that case, let’s try the “easy” way.

Matlab sparse matrices are among the classes that Matlab cannot pass to Python. The workaround requires enough RAM to hold the full matrix to pass to and from Python.

1. convert Matlab sparse to full
2. process sparse data in Python
3. convert Python sparse to full

All commands are issued in Matlab.

A = sparse(eye(5));  % test data

As = py.scipy.sparse.csc_matrix(full(A))

results in:

As =
  Python csc_matrix with properties:

                   dtype: [1×1 py.numpy.dtype]
    has_canonical_format: 1
      has_sorted_indices: 1
                     nnz: [1×1 py.int]
                   shape: [1×2 py.tuple]
                maxprint: [1×1 py.int]
                  indptr: [1×1 py.numpy.ndarray]
                 indices: [1×1 py.numpy.ndarray]
                    data: [1×1 py.numpy.ndarray]

      (0, 0)	1.0
      (1, 1)	1.0
      (2, 2)	1.0
      (3, 3)	1.0
      (4, 4)	1.0

PyGame is trivial to install on most devices. PyGame allows playing sound from Numpy arrays.

pip install pygame

pip prefers .whl binary wheel instead of compiling from source when a wheel is available.

Source compile pip (optional)

This is not usually needed

apt install git ffmpeg libsdl-image1.2-dev libsdl-mixer1.2-dev libsdl-ttf2.0-dev libsdl1.2-dev libfreetype6-dev libavformat-dev libportmidi-dev libswscale-dev

git clone https://github.com/pygame/pygame/

pip install -e pygame

Notes

• Most packages are available from pip as wheel .whl.
• Pygame Install reference

Matlab or GNU Octave can create lossless videos from image stacks or figure series by either:

• Save PNG image stack, then losslessly convert PNG stack to video
• Save from Matlab directly as lossless JPEG2000 video

This example saves lossless MJPEG2000 .mj2 video plots directly from Matlab VideoWriter.

v = VideoWriter('test.mj2', 'Archival');
v.LosslessCompression = true;
v.FrameRate = 5;  % arbitrary
open(v)

f = figure;
% or axes

pause(0.2)
% let plot initially draw

for i = 1:.05:3;

    line(0:.01:1, (0:.01:1).^i)
    % dummy plot sequence

    v.writeVideo(getframe(f))
end

close(v)

Play Motion JPEG 2000 .mj2 with FFmpeg from the command line (outside of Matlab):

ffplay movie.mj2

GNU Octave video package also has a “VideoWriter” function.

Consider the semi-automated script to install Debian to the eMMC of the Beaglebone Black.

This procedure requires a 4 GB micro SD card or larger, and assumes a Linux laptop. We will copy a Linux operating system image to eMMC from a micro SD card.

On laptop PC, download the Debian Beaglebone Black (BBB) image. For an old 2 GB Beaglebone, use the 2gb image.

Type lsblk, note which drives are listed, then insert the SD card into the laptop and type lsblk again–the new item is your SD card. We assume /dev/mmcblk0.

Extract image to SD card:

xz -cd BBB-*.img.xz > /dev/mmcblk0

Extraction takes about 5 to 20 minutes at ~ 5 MB/sec, writing uncompressed ~ 2 GB to the SD card.

[optional] monitor data writing to SD card with

iotop

Ensure writing has completed with

sync

Insert micro SD card into the (non-powered) BBB and then apply power. Beaglebone four onboard LEDs flash back and forth in a “cylon” or “knight rider” pattern. During this time, the micro SD card program is flashing the onboard eMMC. With an FTDI to USB adapter fit onto J1, the process can be monitored via the screen program. The automatic flashing to eMMC process should complete in about 10-20 minutes.

Once the BBB has shut down, REMOVE the micro SD card from the BBB. Reset the power or push the onboard POWER button next to the Ethernet jack.

Boot from eMMC

There is an SSH server running by default available through the mini-USB port.

Find the LAN IP address of the BBB (plug the BBB into the Ethernet). Assuming local network IP addresses 192.168.1.xxx, from a laptop:

nmap -p 22 192.168.1.* --open

This command lists SSH servers on the network, so do it once with the BBB unplugged from the network, then again with the BBB plugged in. If nmap isn’t available, the findssh program uses plain Python to find SSH or other servers.

Assuming BBB is at 192.168.0.5, from a laptop:

ssh debian@192.168.0.5

Notes

If timezone or locales issues try on Beaglebone:

dpkg-reconfigure tzdata
apt install locales
dpkg-reconfigure locales

Troubleshoot HDMI with parse-edid, obtained by:

apt install read-edid