Scientific Computing

Convert animated GIF to PNG stack using ImageMagick by:

magick in.gif out_%04.png

where 04 is governed by the number of images in the GIF–04 accommodates up to 10000 images.

GIFs are not a great format for science image data, because the palette is compressed to 8-bit (256 colors). For plotting reduced data, GIFs can be fine.

Spyder IDE is a complex but usually stable Python program. A problem symptom is Spyder not getting past the splash logo or not even showing the splash logo.

To totally reset Spyder (erasing all user preferences for Spyder), type in Terminal / Command Prompt:

spyder --reset

Normally, that fixes Spyder. To diagnose further, start Spyder from Terminal instead of OS Start menu, it might give some hints.

Access GPU CUDA, cuDNN and NCCL functionality are accessed in a Numpy-like way from CuPy. CuPy also allows use of the GPU in a more low-level fashion as well.

Before starting GPU work in any programming language realize these general caveats:

• I/O heavy workloads may make realizing GPU benefits more difficult
• Consumer GPUs (GeForce) can be > 10x slower than workstation class (Tesla, Quadro)

CUDA requires a discrete Nvidia GPU. Check for existence of an Nvidia GPU by:

• Linux: a blank response means an Nvidia GPU is not detected.

  lspci | grep -i nvidia

• Windows: Look under the “render” tab to see if an Nvidia GPU exists.

  dxdiag

Determine the Compute Capability of the GPU and install the correct CUDA Toolkit. CuPy is installed distinctly depending on the CUDA Toolkit version installed on your computer. Reboot.

CuPy syntax is very similar to Numpy. There are a large set of CuPy functions relevant to many engineering and scientific computing tasks.

import cupy

dev = cupy.cuda.Device()
print('Compute Capability', dev.compute_capability)
print('GPU Memory', dev.mem_info)

The should return like:

Compute Capability 75

If you get error like

cupy.cuda.runtime.CUDARuntimeError: cudaErrorInsufficientDriver: CUDA driver version is insufficient for CUDA runtime version

This means the CUDA Toolkit version is expecting a newer Nvidia driver. The Nvidia driver can be updated via your standard Nvidia update program that was installed from the factory. “Table 1” of the CUDA Toolkit release notes gives the CUDA Toolkit required Driver Versions.

Examples:

• Python PyCUDA Matrix Multiplication Benchmark matmul_cuda.py
• non-Python Graphics Benchmarks 

Alternatives to CuPy include Numba.cuda, which is a lower-level C-like CUDA interface from Python. CUDA for Julia is provided in JuliaGPU. Anaconda Accelerate was discontinued

A code cell in popular Python IDEs including PyCharm and Spyder is created by line starting with # %%. This “code cell” is analogous to IPython code cells and Matlab code sections.

You will see like

import math

# %% user data
x = 3
y = 4
# %% main loop
for i in range(5):
    x += y

The code cells allow running sections of code in an IDE without the need to constantly set/unset breakpoints in the IDE. They also catch the eye of developers to delineate logical blocks of code in the algorithm.

We encourage the use of code cell syntax, even if you don’t use them in the IDE directly, as the IDE will highlight sections of code to visibly delineate these separate parts of the algorithm.

GitLab Community Edition is open source. Anyone may host their own self-managed GitLab instance if desired instead of gitlab.com. Git SSH. For this example, we use Kitware’s CMake GitLab instance.

First, create an account on the self-managed GitLab instance and fork the desired repo. This will be available like

git clone https://gitlab.kitware.com/username/cmake

To git push using SSH, type:

git config --global url."ssh://gitlab.kitware.com/".pushInsteadOf https://gitlab.kitware.com/

Generate an SSH key–don’t reuse SSH keys between sites.

ssh-keygen -t ed25519 -f ~/.ssh/kitware

Go to the GitLab SSH Key page and add the contents of ~/.ssh/kitware.pub

Add to ~/.ssh/config:

Host gitlab.kitware.com
  User git
  IdentityFile ~/.ssh/kitware

Now checkout a new branch, make your changes according to project guidelines and submit a merge request.

xarray.open_dataset can open HDF5 files. However, unexpected HDF5 file layouts can cause Python to quietly crash without error message. This is true even with the minimum required versions of xarray, h5py and h5netcdf installed.

We don’t have a specific workaround for this other than to use h5py to build up an xarray Dataset variable-by-variable.

• GitHub API v4 does not include Wikis.
• GitLab API v4 does include Wikis

To duplicate a GitHub repo AND the GitHub Wiki benefits from scripting.

An example solution is in GitEDU. This requires manually clicking to enable the new wiki via a web browser for each repo wiki.

GitHub Wiki Git accessibility

In general, GitHub Wiki is just another Git repo. The URL for the GitHub Wiki is obtained by appending .wiki.git to the associated GitHub repo URL.

Example:

• Main repo: github.invalid/username/reponame
• Wiki repo: github.invalid/username/reponame.wiki.git

Related: Moving GitHub Wiki

GitHub Wikis are not accessible via GitHub API v4. We can use a couple simple Git command line statements to move a GitHub Wiki.

For these examples, we assume that:

• old wiki: github.invalid/username/repo1.wiki.git
• new wiki: github.invalid/username/repo2.wiki.git

Copy GitHub Wiki to the laptop:

git clone --bare https://github.invalid/username/repo1.wiki.git

Browse to new Wiki and create blank Wiki

Mirror push Wiki to new repo:

git -C repo1 push --mirror https://github.invalid/username/repo2.wiki.git

Once you see the new Wiki is OK, remove the old Wiki pages if desired:

git -C repo1 rm *.md

git -C repo1 commit -am "deleted old wiki pages"

git -C repo1 push

Related: Duplicating GitHub Wiki

One should almost always use a Python context manager when working with file I/O in Python. Context managers for Python I/O resources help avoid exceeding system resource limits. For long running jobs, context managers help avoid random crashes due to excess file I/O resource utilization from files left hanging open. There are edge cases where you do need to keep the handle open without context manager–for example, inside a for loop. In many cases, it may be better and easier to let the file open and close with the context manager.

It is also possible to create your own content managers with Python contextlib, which we use in georinex for example.

Context Manager examples: assuming:

from pathlib import Path

fn = Path('~/mydir/myfile').expanduser()

simple file I/O:

with fn.open('r') as f:
    line = f.readline()

Note, if just reading a whole file, consider pathlib.Path methods like:

txt = fn.read_text()

b = fn.read_bytes()

h5py:

import h5py

with h5py.File(fn, 'r') as f:
    data = f['myvar'][:]

NetCDF4:

import netCDF4

with netCDF4.Dataset(fn, 'r') as f:
    data = f['myvar'][:]

The Free TightVNC server works with Ubuntu and other modern Linux distros. In general the default 3-D Ubuntu desktop is not available over VNC, so we use a traditional 2-D desktop environment over VNC.

Install Linux VNC Server:

apt install tightvncserver openbox

Choose desktop environment on server PC: XFCE4, Openbox or other you prefer. Pick ONE of the following:

Install XFCE on server PC

apt install xfce4
apt remove xscreensaver xscreensaver-data

Create ~/.vnc/xstartup

#!/bin/sh
unset SESSION_MANAGER
startxfce4 &

Correct possible keystroke issues on server PC:

xfconf-query -c xfce4-keyboard-shortcuts -p /xfwm4/custom/'<'Super'>'Tab -r

Openbox is the lightest-weight desktop environment–right click to open a menu. Otherwise, you just have a plain gray background, extremely minimal–good for embedded systems and old PCs.

Put into file ~/.vnc/xstartup

#!/bin/sh
unset SESSION_MANAGER
exec openbox-session &

Start VNC server

vncserver :1 -geometry 1200x700 -localhost

Create a file ~/startVNC.sh containing:

#!/bin/sh
vncserver :1 -geometry 1200x700 -localhost

and then from the server username

crontab -e

adding the line:

@reboot  /home/username/startVNC.sh

If you have an encrypted /home drive, VNC and SSH require configuration to allow decrypting home drive upon SSH login.

Setup VNC client

On your laptop, install VNC client

apt install tigervnc-viewer

Alternatively:

apt install vncviewer

Create a shell script:

#!/usr/bin/env bash
ssh -f -L 5901:localhost:5901 user@IPaddress sleep 1;
vncviewer  localhost::5901

Notes

Openbox-Message: Unable to find a valid menu file “/var/lib/openbox/debian-menu.xml”

When experiencing this error, I’m also unable to open a terminal.

• You can leave your VNC desktop running – it is not the same as your local desktop.
• It is a little tricky to share your local desktop reliably. X11VNC can be more trouble than it is worth. It’s MUCH easier to startup a new separate desktop session with vncserver or x11vnc
• After installing a new desktop environment, at your next local login, you’ll need to rechoose the traditional Ubuntu desktop (it will then remember this choice).

Security

Be sure that ports 5900-5999 are NOT exposed to outside world–VNC is NOT secure by itself! VNC must be tunneled over the Internet with SSH.

Ports exposed by the PC to the network are listed by:

ufw status

You should see only port 22 and any other ports only if you know what they are for.

Kill a frozen/undesired desktop by logout of VNC from your laptop, then:

vncserver -kill :1

Fix non-working right-click Openbox menu by creating ~/.config/openbox/menu.xml with the content

<?xml version="1.0" encoding="utf-8"?>
<openbox_menu xmlns="https://openbox.org/" xmlns:xsi="https://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="/usr/share/openbox/menu.xsd">
    <menu id="root-menu" label="Openbox 3">
        <item label="Run Program">
            <action name="Execute">
                <execute>
                    gmrun
                </execute>
            </action>
        </item>
        <separator/>
        <item label="Terminal">
            <action name="Execute">
                <execute>
                    xterm
                </execute>
            </action>
        </item>
  </menu>
</openbox_menu>

Fix missing XFCE4 VNC menu icons under: Settings → Appearance. Set: Style-Xfce, Icons-ubuntu-mono-light, Fonts-turn hinting on if you want