Scientific Computing

Advanced FreeRDP options greatly improve RDP Windows Remote Desktop connections over weak internet connections.

A Linux laptop can connect to Windows PCs via SSH port forwarding using:

#!/bin/sh

ssh -f -L 4389:localhost:3389 remoteusername@remoteIP sleep 1;

xfreerdp /cert-ignore /v:localhost:4389 \
/bpp:8 /network:modem /compression -themes -wallpaper \
/clipboard /audio-mode:1 \
/auto-reconnect -glyph-cache

/clipboard

enable bidirectional clipboard

/bpp:8

uses 256 colors-low the quality, but really makes a speed improvement. Might not display videos (e.g. VLC)–try /bpp:16 if trouble.

/bpp:16

uses 65536 colors, saving bandwidth over 24-bit color with negligible visible difference for most basic uses.

/network:modem /compression

reduce bandwidth via compression (trade CPU usages for network bandwidth)

-themes -wallpaper

great speedup by not needlessly sending background graphics repeatedly

/async-update /async-input

disable RDP waiting for screen updates to reach you before it accepts input. These allow clicking ahead before the screen updates. Be careful of clicking unwanted options while using the PC.

-glyph-cache

disable glyph caching. Note: this can cause garbled characters and radio boxes.

/audio-mode:1

disable FreeRDP audio redirection (do not play sound from remote PC)

/auto-reconnect

automatically reconnect on failure (also works over SSH tunnel)

Notes

For remotely operated Digital amateur radio modes, it’s important to keep the audio generation/reception on the remote PC.

Related: RDP over SSH port forwarding

Usually it’s desired to check Python type annotations for an entire project. However, when introducing type checking to a large Python project, it can be beneficial to introduce type checking gradually, to avoid massive refactoring all at once. You will almost certainly find old code bugs due to type checking that will require refactoring, which introduces risk. Mitigate this risk by introducing type checking gradually.

MyPy is a strongly recommended type checker, though others exist. Use pyproject.toml to configure MyPy default behavior. To tell MyPy to only check certain files, use the MyPy –follow-imports= option like:

mypy --follow-imports=skip myproj/foo.py myproj/bar.py

Only certain directories can be checked like:

mypy --follow-imports=skip myproj/sub/ myproj/sub2

Once the type checks pass via mypy --follow-imports=skip, we recommend trying

mypy --follow-imports=silent

to improve robustness of the type check for those files / directories.

Enhanced mypy usage

32-bit Python binaries can run on 64-bit OS. To precisely detect if the operating system is 64-bit or 32-bit, check if Python itself is 32-bit to avoid falsely detecting a 64-bit OS as 32-bit. This check is just for the Python interpreter. To detect OS parameters in a cross-platform-robust way further checks are necessary.

python -c "import sys; print(sys.maxsize > 2**32)"

• 32-bit Python will print False
• 64-bit Python will print True

In a script:

import sys

def is_64bit() -> bool:
    return sys.maxsize > 2**32

If the history of a Git repository has been rewritten, but someone has forked the repository before the history was rewritten, they will be unable to merge due to the unrelated history. This method corrects the history in their forked branches to match the original repository.

In this synthetic example:

• Original Git repository: github.com/username/coolprog main branch develop
• Forked Git repository: github.com/jane/coolprog
• A colleague, Jane, with GitHub username jane, has created a branch from develop named feature1. However, scivision changed the history of develop after the fork.

The “easy” way can reduce the risk of compromising months or years of work compared to the “Pure Git” way.

Easy way

If the “Pure Git” way was attempted first, avoid using that directory. Instead, git clone the forked repository again.

Clone the forked repository into a temporary directory:

cd ${TMPDIR}
git clone https://github.invalid/jane/coolprog

cd coolprog
git switch feature1

Create a branch jane-feature1 in the original repository to copy the forked changes:

cd ~/code/coolprog

git switch -c jane-feature1

Manually merge the changes made in the feature1 branch using compare folders.

Note that the repository will likely have many files that are not in the fork due to .gitignore.

For any new files that need to be tracked by Git: git add filename

When satisfied with the manual merge, commit the changes in jane-feature1 to give proper credit to jane:

GIT_COMMITTER_NAME="Jane" GIT_COMMITTER_EMAIL="jane@users.noreply.github.com" git commit --author="Jane <jane@users.noreply.github.com>"

Note that Jane’s real-life email is not used to avoid exposing it to spammers. Verify the commit with git log.

Upload the changes to the Git repository:

git push -u origin jane-feature1

Merge changes into main branch

To incorporate these new features into the develop branch:

git switch develop

git merge jane-feature1

git push

Jane should create a new fork of the repository, deleting the old fork with the incorrect history to avoid repeating this process for future changes.

Reforking current repository

Jane should refork from the original GitHub repository with the corrected history to clean up the “unrelated changes” issue.

To do this, assuming no further changes have been made:

1. Delete the coolprog directory on the local machine.
2. “Delete repository” at the bottom of https://github.invalid/jane/coolprog/settings.
3. Fork the original repository again from https://github.invalid/scivision/coolprog#fork-destination-box.

This procedure is typically unnecessary and is only required when Git history has been rewritten before the fork.

Pure Git

This method requires Jane to force push and execute these commands (or grant write access to the forked repository). It is risky, so the “Easy” way is recommended.

Clone the forked repository into a temporary directory:

cd ${TMPDIR}
git clone https://github.invalid/jane/coolprog

cd coolprog
git switch develop

Add the original repository as upstream of the fork:

git remote add upstream https://github.invalid/scivision/coolprog
git fetch upstream

git rebase upstream/develop

git switch feature1
git rebase develop

If these changes are successful and the Git history is confirmed to be correct, a git push -f can be performed. Ensure secure, independent copies of the GitHub repository are available, as force-pushing overwrites Git history and may erase work.

Sometimes for Git repos with a long, complicated history this doesn’t work, and would be time consuming to fix. In that case, let’s try the “easy” way.

Matlab sparse matrices are among the classes that Matlab cannot pass to Python. The workaround requires enough RAM to hold the full matrix to pass to and from Python.

1. convert Matlab sparse to full
2. process sparse data in Python
3. convert Python sparse to full

All commands are issued in Matlab.

A = sparse(eye(5));  % test data

As = py.scipy.sparse.csc_matrix(full(A))

results in:

As =
  Python csc_matrix with properties:

                   dtype: [1×1 py.numpy.dtype]
    has_canonical_format: 1
      has_sorted_indices: 1
                     nnz: [1×1 py.int]
                   shape: [1×2 py.tuple]
                maxprint: [1×1 py.int]
                  indptr: [1×1 py.numpy.ndarray]
                 indices: [1×1 py.numpy.ndarray]
                    data: [1×1 py.numpy.ndarray]

      (0, 0)	1.0
      (1, 1)	1.0
      (2, 2)	1.0
      (3, 3)	1.0
      (4, 4)	1.0

PyGame is trivial to install on most devices. PyGame allows playing sound from Numpy arrays.

pip install pygame

pip prefers .whl binary wheel instead of compiling from source when a wheel is available.

Source compile pip (optional)

This is not usually needed

apt install git ffmpeg libsdl-image1.2-dev libsdl-mixer1.2-dev libsdl-ttf2.0-dev libsdl1.2-dev libfreetype6-dev libavformat-dev libportmidi-dev libswscale-dev

git clone https://github.com/pygame/pygame/

pip install -e pygame

Notes

• Most packages are available from pip as wheel .whl.
• Pygame Install reference

Matlab or GNU Octave can create lossless videos from image stacks or figure series by either:

• Save PNG image stack, then losslessly convert PNG stack to video
• Save from Matlab directly as lossless JPEG2000 video

This example saves lossless MJPEG2000 .mj2 video plots directly from Matlab VideoWriter.

v = VideoWriter('test.mj2', 'Archival');
v.LosslessCompression = true;
v.FrameRate = 5;  % arbitrary
open(v)

f = figure;
% or axes

pause(0.2)
% let plot initially draw

for i = 1:.05:3;

    line(0:.01:1, (0:.01:1).^i)
    % dummy plot sequence

    v.writeVideo(getframe(f))
end

close(v)

Play Motion JPEG 2000 .mj2 with FFmpeg from the command line (outside of Matlab):

ffplay movie.mj2

GNU Octave video package also has a “VideoWriter” function.

Consider the semi-automated script to install Debian to the eMMC of the Beaglebone Black.

This procedure requires a 4 GB micro SD card or larger, and assumes a Linux laptop. We will copy a Linux operating system image to eMMC from a micro SD card.

On laptop PC, download the Debian Beaglebone Black (BBB) image. For an old 2 GB Beaglebone, use the 2gb image.

Type lsblk, note which drives are listed, then insert the SD card into the laptop and type lsblk again–the new item is your SD card. We assume /dev/mmcblk0.

Extract image to SD card:

xz -cd BBB-*.img.xz > /dev/mmcblk0

Extraction takes about 5 to 20 minutes at ~ 5 MB/sec, writing uncompressed ~ 2 GB to the SD card.

[optional] monitor data writing to SD card with

iotop

Ensure writing has completed with

sync

Insert micro SD card into the (non-powered) BBB and then apply power. Beaglebone four onboard LEDs flash back and forth in a “cylon” or “knight rider” pattern. During this time, the micro SD card program is flashing the onboard eMMC. With an FTDI to USB adapter fit onto J1, the process can be monitored via the screen program. The automatic flashing to eMMC process should complete in about 10-20 minutes.

Once the BBB has shut down, REMOVE the micro SD card from the BBB. Reset the power or push the onboard POWER button next to the Ethernet jack.

Boot from eMMC

There is an SSH server running by default available through the mini-USB port.

Find the LAN IP address of the BBB (plug the BBB into the Ethernet). Assuming local network IP addresses 192.168.1.xxx, from a laptop:

nmap -p 22 192.168.1.* --open

This command lists SSH servers on the network, so do it once with the BBB unplugged from the network, then again with the BBB plugged in. If nmap isn’t available, the findssh program uses plain Python to find SSH or other servers.

Assuming BBB is at 192.168.0.5, from a laptop:

ssh debian@192.168.0.5

Notes

If timezone or locales issues try on Beaglebone:

dpkg-reconfigure tzdata
apt install locales
dpkg-reconfigure locales

Troubleshoot HDMI with parse-edid, obtained by:

apt install read-edid

Some programs simply cannot run without an X11 graphics server available. In other cases, testing a program requires X11 to be available, even though the windows aren’t visible. An example is creating Matplotlib figures in a continuous integration environment.

The X server virtual framebuffer (Xvfb) allows creating non-visible desktop windows without the computation burden of an full X11 graphics environment.

Xvfb is effective for most Linux systems without displays, particularly for headless or embedded systems including:

• Raspberry Pi
• Windows Subsystem for Linux
• continous integration servers
• virtual machines

apt install xvfb

Add to ~/.profile

Xvfb :99 &
export DISPLAY=:99

Upon opening a new Terminal or SSH connection, the X11 virtual framebuffer is available.

Test XVFB by running a graphics-only program, for example:

xmessage -center "Hello World"

If XVFB or X11 isn’t working, the message will be like

Error: Can't open display: :0.0

With the recent release of iOS 13, we are seeing Apple iOS devices that suddenly won’t connect to any authenticated WiFi. The solution seems to be to do one more reboot of the iOS device and if needed, “forget” and re-add the desired WiFi system. Rebooting the WiFi router is generally NOT needed.