Scientific Computing

IJulia allows running Julia from within the web browser-based Jupyter IDE. In general (for all operating systems) it’s recommended to install and update Julia via the downloads from Julia website.

Install Jupyter: locate Jupyter binary location

• Windows: where jupyter
• Linux, macOS: which jupyter

If Jupyter is not installed, just do

conda install jupyter

Install IJulia: start Julia with this environment variable (only necessary once)

• Linux / macOS:

  JUPYTER=$(which jupyter) julia

• Windows:

  $Env:JUPYTER=$(where.exe jupyter)
  
  julia

Finally, type at Julia prompt:

using Pkg

Pkg.add("IJulia")

This installs numerous packages via conda automatically.

Existing .jl files are NOT runnable from IJulia. Start the Notebook by EITHER:

• Terminal/Command Prompt: jupyter lab
• Julia: using IJulia; notebook()

Create an interactive Julia .ipynb Julia notebook by clicking New → Julia.

• IJulia install reference

Some Raspberry Pi models such as the Zero and Zero W do not have an Ethernet port on the board. While one can use a USB-Ethernet adapter in the USB OTG port, if one wants to use only the Pi itself without adapters, the procedure below is required.

Install Raspberry Pi operating system on micro SD card.

On the SD card, edit /boot/config.txt, adding the line:

dtoverlay=dwc2

Now pick one of the following connection methods.

Ethernet over USB

Edit boot/cmdline.txt, adding after rootwait on the same line with a space:

rootwait modules-load=dwc2,g_ether

Boot the Pi with the micro SD card inserted, waiting 90 seconds or so. Then type from laptop (username, hostname are those picked for the Pi by the Raspberry Pi Imager program):

ssh username@hostname.local

If this doesn’t work, ensure that you see the new Ethernet port on your laptop. On Linux this would be seen in

ip a

reference

serial over USB

This method uses very basic USB drivers that should be on any laptop operating system.

Edit boot/cmdline.txt, adding after rootwait on the same line with a space:

rootwait modules-load=dwc2,g_serial

Boot the Pi with the micro SD card (using an HDMI monitor or SSH), and type in the Pi:

systemctl enable getty@ttyGS0.service

this is a one-time command that will be “remembered”

Reboot the Pi and connect from your laptop with a serial client like PuTTY at 115200 baud. You can find the port the device is on in Linux from

dmesg

ls /dev/tty*

before and after plugging in the Pi.

reference

Related: Headless Raspberry Pi setup with Ethernet

Since PyPI accepts Markdown formatted README.md, there is less reason to use the more complicated syntax of ReStructured Text .rst files. We have converted hundreds of README.rst to README.md with the process below.

Bulk convert RST .rst to Markdown .md:

pandoc -f rst -t markdown README.rst -o README.md

Slight hand correction: remove most \ that were inserted (find and replace with nothing)

Replace in Git repo:

git rm README.rst
git add README.md

For Python repos, be sure that pyproject.toml is configured for README.md

Eliminating unnecessary (particularly large) files and removing needless historical development details are two significant parts of preparing a Git repo for public release. The general public users, even if of a limited group don’t need large amounts of code development history, probably littered with large files. Here are several straightforward steps to prepare code for public Git release.

1. Create an empty Git repo

GitHub is an obvious first choice, as GitHub has by far the largest number of users and excellent integration with third party tools. Bitbucket and GitLab are two worthy alternatives.

Create an empty Git repo at the website, then clone the empty repo you created to the computer.

git clone https://github.invalid/username/myrepo

Copy the files you want into the myrepo directory–we’ll clean up extra files next

2. Remove unneeded files

These commands are executed in the new myrepo directory you cloned, NOT your old directory. These commands assume a Unix-like shell.

Find the biggest directories in myrepo directory:

du -h | sort -h

To inspect biggest files within ~/mydir:

ls -h mydir | sort -h

Find binary files (non-text, non-code) recursively:

find . -type f | perl -lne 'print if -B'

Find and eliminate .DS_Store files (from macOS):

find . -name .DS_Store

and then add .DS_Store to .gitignore

Sometimes it’s handy to remove or list all files EXCEPT those matching a pattern (inverse globbing):

shopt -s extglob

ls !(*inverse_pattern*)

Keep unwanted files out of the Git repo in the future by adding filename, directory names, and globbing patterns to .gitignore

3. Share and collaborate

When you’re confident things are ready, do

git add .

git commit -am "initial public release"

git push

and your files are on the Web for all.

Users will use GitHub Issues and Pull Requests to request and suggest code changes.

Instead of adding Collaborators, start by having people who want to make changes Fork and then Pull Request.

4. Ensure quality

Continuous Integration is vital to maintaining and improving code quality. GitHub Actions CI is a popular choice.

Linux Google Earth Pro is available at no charge. Download Google Earth 64-bit .deb or direct link and install:

gdebi google-earth-pro-stable*amd64.deb

Open from Google Earth menu icon, or from Terminal:

google-earth-pro

which is a softlink under /usr/bin/ to /opt/google/earth/pro/googleearth

This also installs a software source in /etc/apt/sources.list.d/google-earth-pro.list for automatic updates.

As noted in Google Earth release notes, the key improvements include:

• all Google Earth installs are now “Pro” for free
• high-resolution (Hi DPI) support
• fixed broken networking system (blank links) etc.
• works with new Google Photos layer (that replaced Panoramio)

Matplotlib pcolormesh() is 10-100x or more faster than pcolor(), especially when using cartopy. However, the mesh generation requires valid edge coordinates–NaN is not allowed. A workaround for certain scenarios like geospatial plots is to “smear” the last valid x,y (say, latitude, longitude) out to replace the NaN’s. Like pcolor(), this method hides the invalid values. There may be slight aberrations at the edges.

Example code for pcolormesh with NaN coordinates: pcolormesh_NaN.py

Some Raspberry Pi modles have Bluetooth and 2.4 / 5 GHz 802.11ac WiFi built in or can use a USB dongle for WiFi and Bluetooth. Raspberry Pi and many other embedded single board computers can connect to the internet via Bluetooth WPAN a.k.a. tethering or connection sharing. This method avoids tying up the WiFi connection of a phone, or putting sensitive WiFi credentials on an embedded device. This method inherently limits internet bandwidth to the embedded device to about 1.6 Mbps practical throughput due to the legacy Bluetooth WPAN specification.

NetworkManager can be used from the command line “nmcli” to make the Bluetooth internet connection. Create a script and “@reboot” cron job to make a semi-permanent WPAN connection. Connect to the internet host device e.g. smartphone with Bluetooth tethering turned on Bluetooth UUID like:

nmcli device connect AA:BB:CC:DD:EE:FF

It’s also possible to configure USB tethering to the phone; I’ve done that via the NetworkManager GUI from the desktop environment.

nmcli examples

BlueZ5 allows using Bluetooth on Linux for useful tasks like:

• playing sound from the Linux device over Bluetooth headphones
• send text with Android phones

This works for Linux computers in general, from laptops to Raspberry Pi or other single board computers with Bluetooth on-board or via USB adapter. BlueZ is very handy for command line Bluetooth on Linux:

apt install bluez5

From any Linux computer with Bluetooth, pair the Bluetooth device from Terminal:

Hardware enable Bluetooth adapter

rfkill unblock bluetooth

start HCI UART service

systemctl start hciuart

Go into Bluetooth interactive mode. Use sudo if “no default controller available” error occurs.

bluetoothctl

shows a list of devices already paired, and goes into an interactive Bluez program.

show Bluetooth adapters usable (there must be at least one).

list

software enable Bluetooth adapter

power on
agent on

Find Bluetooth devices to pair with

scan on

A list of Bluetooth pairing-mode UUIDs stream in continuously. Wait several seconds if you don’t immediately see your device. Be sure the device (e.g. smartphone/laptop) is in visible/pairing mode.

Stop the scan:

scan off

Pair with Bluetooth device, where uuid is your device

pair uuid

trust uuid

Connect to the Bluetooth device:

connect uuid

bluetoothctl commands: from within bluetoothctl interactive prompt.

• paired-devices: list devices this computer is paired with
• info UUID: for device with UUID, list capabilities (Bluetooth service profiles).

For example, PANU is PAN user.

Here are several basic Bluetooth commands from the Linux terminal.

List installed Bluetooth devices from Linux:

rfkill list

The listing will look like:

0: phy0 Bluetooth: bluetooth Soft blocked: yes Hard blocked: no

enable Bluetooth devices:

rfkill unblock bluetooth

Which reveals another device:

1: hci0: bluetooth Soft blocked: no Hard blocked: no

List Bluetooth adapter (device) UUID

hcitool dev

Which returns:

hci0

and Bluetooth adapter (device) UUID.

enable Bluetooth

hciconfig hci0 up

BlueZ interactive interface

bluetoothctl

This enables pairing/trusting devices, setup Linux Bluetooth audio, and much more.

Related:

• setup Linux Bluetooth audio from Terminal
• send Bluetooth Low Energy Beacons / iBeacon

Pytest cleanly handles many continuous integration issues. Pytest is worthwhile to setup tests and improve test coverage. It’s straightforward to incrementally upgrade from obsolete nose and legacy unittest modules to Pytest.

A frequent CI task is comparing floating point numbers. Computer representations of floating point numbers have finite precision. Real numbers associativity doesn’t completely apply to digital floating point representations. A practical solution to this problem is to compare numbers (scalars or arrays) to within an absolute and relative tolerance. Widely known functions exist to compare “equality” of floating point numbers in Python and Fortran real/assert.F90 among other numerical languages.

pytest.approx provides a syntactically clean approach that may be more intuitive and readable for CI applications. It works with scalars and arrays of all sorts including the ubiquitous numpy.ndarray.

pytest.approx() can be used to compare Numpy ndarrays, lists, tuples, and scalars. However, for Numpy ndarrays, a richer feature set and text output is available with numpy.testing.assert_allclose().

import numpy as np
from pytest import approx


def test_mynums():
    x = np.array([2.00000000001, 1.99999999999])

    assert x == approx(2.)
# or
    np.testing.assert_allclose(x, 2.)

NaN is an unrepresentable, undefined quantity as results from invalid computations like:

• ∞ / ∞
• 0 / 0

NaN are also useful as sentinel values to indicate a problem with a computation for example. Python sentinel values include NaN and None. In Julia, performance is approximately the same for nan vs. nothing sentinel values.

The option pytest.approx(nan_ok=True) allows one or more NaN’s in the test array.

Absolute and relative tolerance are important considerations for CI, particularly across OS and computer types, where order of operation and numerical CPU/GPU/APU/etc. tolerances and low level math routines (BLAS, AVX, etc.) differ. Especially (perhaps obviously) for single precision arithmetic vs. double precision arithmetic, tolerances will be larger. pytest.approx() default tolerances work for many situations. Absolute tolerance shouldn’t be 0 to avoid unexpected effects near 0. Depending on the parameter, rel=1e-4 may be more appropriate for CI tests that need to work on a variety of systems.

YouTube screen capture with audio is simple, cross-platform and stable using nothing more than FFmpeg.

ffmpeg \
-video_size 1024x720 \
-framerate 10 \
-f x11grab -i :0.0+100,200 \
-f pulse -ac 2 -i default \
-c:a libmp3lame -ar 48000 \
out.mkv

The audio config is handled by:

-f pulse -ac 2 -i default

send stereo audio with the default PulseAudio device.

DO NOT USE -ac 1 even with monaural audio as you’ll get lots of errors like

non-monotonous DTS in output stream 0:1;

-c:a libmp3lame -ar 44100

encode audio as MP3 with a 44.1 kHz sampling frequency (passing audio up to about 22 kHz, which is CD-quality).

List PulseAudio devices:

pactl list sources

Control audio sources with:

pavucontrol

Video config -framerate 10 -f x11grab -i :0.0+0,0 screengrab at 10 fps, starting in the upper left hand corner, send a 1024x720 pixel video. If your screen resolution is more than 1024x720, there would be unsent regions at the bottom and/or right side.

• FFmpeg desktop capture
• FFmpeg compilation
• YouTube recommended video parameters

Related: YouTube Live streaming with FFmpeg