Scientific Computing

sudo XWayland apps

When using XWayland apps, sudo synaptic or other GUI programs may result in:

No protocol specified. Unable to init server **: cannot open display: :0.

A workaround for the GUI sudo permissions issue with Wayland is adding to ~/.profile

xhost +si:localuser:root > /dev/null

NOTE: this defeats non-root security advantages of Wayland over X11.

reference

Stellarium scripting engine

Citizen science images of aurora and celestial features can often be noisy. Additionally, consumer and even prosumer cameras manipulate images in ways that typically cannot be completely disabled or even easily quantified in all cases. To make scientific use of images, the image metadata must include:

  • Geographic coordinates (e.g. from GPS)
  • time of image (accuracy ~ 10 seconds for wide angle view, ~ 1 second with < 20 degree FOV).

Stellarium helps manual verification of image calibration. Stellarium can also be used from the web browser without needing any install or plugins. F11 toggles full screen mode.

Press F12 to toggle Stellarium “scripts” menu.

Stellarium can use ECMAScript, which is like a generalized, formal JavaScript. Scripts have a .ssc or .inc filename extension.

We provide several example Stellarium scripts.

Mayavi Python easy install

Mayavi may be thought of as a Python layer atop VTK, making common 3-D data plotting tasks easy. Mayavi is installed via pip or conda. VTK, Traits, et al have .whl binary wheels, which avoid the previously painful build process.

Because of the large number of prereq packages for Mayavi, we strongly urge installing Mayavi in a seperate virtualenv or Conda environment.

conda install mayavi

Mayavi makes high quality manipulable volume plots. Create a file scalar_field.py with the content

from mayavi import mlab
import numpy as np

x, y, z = np.mgrid[-10:10:20j, -10:10:20j, -10:10:20j]
s = np.sin(x*y*z)/(x*y*z)

scf = mlab.pipeline.scalar_field(x,y,z,s)
mlab.pipeline.volume(scf)
mlab.show()

Test programs for Mayavi. Mayavi should not error with

ImportError: Could not import backend for traits

There may be spurious errors during install such as:

ModuleNotFoundError: No module named ‘vtk’

but Mayavi usually works anyway.

Geospace preprint archives

While ArXiv is among the earliest and best known preprint archives, more focused archives can provide easier access to a targeted audience with good reputation. Here are a few I’ve come across relevant to geosciences:

Login show load avg, free memory, free disk

Display a login message using a message of the day (MOTD) with this script under /etc/update-motd.d/

The login message you’ll get will include:

Last Boot..........: 2020-04-12 14:29:29
Memory.............: 822 MB (Available) / 1021 MB (Total)
Load Averages......: 0.03, 0.08, 0.10 (1, 5, 15 min)
Running Processes..: 12
Free Disk Space....: 81 GB of 253 GB on /

Python script for MOTD:

#!/usr/bin/env python3
import sys
import psutil
from datetime import datetime
import shutil
from pathlib import Path

lastboot = datetime.fromtimestamp(psutil.boot_time())
vmem = psutil.virtual_memory()
drv = Path('~').expanduser().anchor
du = shutil.disk_usage(drv)

print("Last Boot..........:", lastboot)
print(f"Memory.............: {vmem.available//1000000} MB (Available) /  {vmem.total//1000000} MB (Total)")
if sys.platform == "linux":
    print(f"Load Averages......: {psutil.getloadavg()} (1, 5, 15 min)")

print("Total Processes....:", len(psutil.pids()))
print(f"Free Disk Space....: {du.free//1000000000} GB of {du.total//1000000000} GB on {drv}")

Ubuntu uses Python script “/usr/bin/landscape-sysinfo” to print a similar MOTD on login.

Ninja job pools for low memory CMake builds

An increasing number of systems have multiple CPUs, say four, six or eight but may have modest RAM of 1 or 2 GB. An example of this is the Raspberry Pi. Ninja job pools allow specifying a specific limit on number of CPU processes used for a CMake target. That is, unlike GNU Make where we have to choose one CPU limit for the entire project, with Ninja we can select CPU limits on a per-target basis. That’s one important benefit of Ninja for speeding up builds of medium to large projects, and why we see increasing adoption of Ninja in prominent projects including Google Chrome. This is another reason why we generally strongly encourage using Ninja with CMake.

Specifically, CMake + Ninja builds can limit CPU process count via target properties:

The global JOB_POOLS property defines the pools for the targets.

Upon experiencing build issues such as SIGKILL due to excessive memory usage, inspect the failed build step to see if it was a compile or link operation, to determine which to limit on a per-target basis.

Example

Suppose that 500 MB of RAM are needed to compile a target and we decide to ensure at least 1 GB of RAM is available to give some margin. Thus we constrain the number of CPU processes for that target based on CMake-detected available physical memory. The appropriate parameters for your project are determined by trial and error. If this method still is not reliable even with a single CPU process, then a possible solution is to cross-compile, that is to build the executable on a more capable system for this modest system.

CMakeLists.txt includes:

set_property(GLOBAL PROPERTY JOB_POOLS one_jobs=1 two_jobs=2)

cmake_host_system_information(RESULT _memfree QUERY AVAILABLE_PHYSICAL_MEMORY)

add_library(big big1.c big2.f90)
if(_memfree LESS 1000)
  set_property(TARGET big PROPERTY JOB_POOL_COMPILE one_jobs)
endif()

Related: tell CMake to use Ninja

Visual Studio update Ninja build

The Ninja build executable for Visual Studio location can be determined from the Visual Studio terminal:

where ninja

The factory Visual Studio Ninja version may be too old for use with CMake Fortran projects. If needed, replace the Visual Studio Ninja executable with the latest Ninja version, perhaps with a soft link to the ninja.exe desired. Add user permission to create symbolic links.

Save figure SVG from Matlab or Matplotlib

Matlab or Matplotlib will save infinite resolution vector graphics SVG format, viewable in web browsers. SVG is usable by LaTeX.

  • vector graphics (SVG or EPS) allow nearly infinite zooming without loss of quality–excellent for line plots and contour plots
  • SVG is viewable by any web browser, and is usable from LaTeX
  • EPS is more commonly used in LaTeX
  • PNG is raster graphics, so has finite (blocky) resolution

Here are examples of saving figures to SVG from Matlab and Matplotlib.

Python

To save figure handle fg, simply do fg.savefig('myfig.svg').

from pathlib import Path
from matplotlib.figure imoprt Figure

fn = Path('~/Documents/mycoolfig.svg').expanduser()

data = [1,2,3,4]

fg = Figure(constrained_layout=True)
ax = fg.gca()
ax.plot(data)

fg.savefig(fn, bbox_inches='tight')

Matlab

Matlab figures in general are saved by exportgraphics.

data = [1,2,3,4]

fg = figure();
plot(data)

exportgraphics(fg, 'matfig.svg')

NetCDF4 segfault on file open

NetCDF4 Fortran library may compile successfully and run for simple programs but segfault on programs where HDF5 is linked directly as well as NetCDF4.

A reason one might directly link both HDF5 and NetCDF is a program that need to read / write files in HDF5 as well as NetCDF format. The symptom observe thus far is the program segfault on nf90_open().

The fix is to compile HDF5 and NetCDF for yourself.