Scientific Computing

If Visual Studio was just installed or upgraded, the Windows SDK may also need to be installed or updated (via the same Visual Studio Installer).

The symptoms of needing Windows SDK install or upgrade with CMake may be upon doing the usual:

cmake -B build

that you get errors like

-- The C compiler identification is MSVC 19.25
-- The CXX compiler identification is MSVC 19.25
-- Check for working C compiler: bin/Hostx64/x64/cl.exe
-- Check for working C compiler: bin/Hostx64/x64/cl.exe - broken
CMake Error at CMakeTestCCompiler.cmake:60 (message):
  The C compiler

    "bin/Hostx64/x64/cl.exe"

  is not able to compile a simple test program.

  It fails with the following output:

    Run Build Command(s):nmake /nologo cmTC_c0691\fast &&       "bin\HostX64\x64\nmake.exe"  -f CMakeFiles\cmTC_c0691.dir\build.make /nologo -L                  CMakeFiles\cmTC_c0691.dir\build
    Building C object CMakeFiles/cmTC_c0691.dir/testCCompiler.c.obj
        bin\Hostx64\x64\cl.exe @AppData\Local\Temp\nmC03F.tmp
    testCCompiler.c
    Linking C executable cmTC_c0691.exe
        "cmake.exe" -E vs_link_exe --intdir=CMakeFiles\cmTC_c0691.dir --rc=rc --mt=CMAKE_MT-NOTFOUND --manifests  -- bin\Hostx64\x64\link.exe /nologo @CMakeFiles\cmTC_c0691.dir\objects1.rsp @AppData\Local\Temp\nmC12A.tmp
    RC Pass 1: command "rc /fo CMakeFiles\cmTC_c0691.dir/manifest.res CMakeFiles\cmTC_c0691.dir/manifest.rc" failed (exit code 0) with the following output:
    The system cannot find the file specifiedNMAKE : fatal error U1077: '"cmake.exe"' : return code '0xffffffff'
    Stop.
    NMAKE : fatal error U1077: '"bin\HostX64\x64\nmake.exe"' : return code '0x2'
    Stop.

Python can control Basler cameras using the official Basler PyPylon API on Linux, macOS, and Windows. The Raspberry Pi and some other ARM systems are also covered by Basler camera drivers.

Install Pylon suite with SDK / Developer options. Setup PyPylon.

Live preview: run Basler Pylon GUI live viewer “PylonViewerApp” or “pylon-viewer” from command line.

If message:

RuntimeError: Payload data has been discarded. Payload data can be discarded by the camera device if the available bandwidth is insufficient.

sudo is needed for Pylon to work in some systems.

Windows users are more likely to have issues with a locked file in one program that was used in another program. This happens in many programs including Python, Matlab and GNU Octave since Windows locks the open file by default.

Matlab symptoms arise from scenarios including:

• HDF5 file that was previously accessed in Matlab
• .m files and folders that were previously edited in Matlab
• other text files opened in Matlab Editor

To fix this issue, try in Matlab or GNU Octave:

fclose('all')

which closes all open files.

If this fails, there could be a bug in Matlab or your code that isn’t closing the file. This can happen when error catching doesn’t have an fclose() inside the catch stanza.

The last resort workaround is:

1. completely close all Matlab sessions
2. move or delete the files that Matlab previously accessed
3. reopen Matlab

When using XWayland apps, sudo synaptic or other GUI programs may result in:

No protocol specified. Unable to init server **: cannot open display: :0.

A workaround for the GUI sudo permissions issue with Wayland is adding to ~/.profile

xhost +si:localuser:root > /dev/null

NOTE: this defeats non-root security advantages of Wayland over X11.

reference

Citizen science images of aurora and celestial features can often be noisy. Additionally, consumer and even prosumer cameras manipulate images in ways that typically cannot be completely disabled or even easily quantified in all cases. To make scientific use of images, the image metadata must include:

• Geographic coordinates (e.g. from GPS)
• time of image (accuracy ~ 10 seconds for wide angle view, ~ 1 second with < 20 degree FOV).

Stellarium helps manual verification of image calibration. Stellarium can also be used from the web browser without needing any install or plugins. F11 toggles full screen mode.

Press F12 to toggle Stellarium “scripts” menu.

Stellarium can use ECMAScript, which is like a generalized, formal JavaScript. Scripts have a .ssc or .inc filename extension.

We provide several example Stellarium scripts.

Mayavi may be thought of as a Python layer atop VTK, making common 3-D data plotting tasks easy. Mayavi is installed via pip or conda. VTK, Traits, et al have .whl binary wheels, which avoid the previously painful build process.

Because of the large number of prereq packages for Mayavi, we strongly urge installing Mayavi in a seperate virtualenv or Conda environment.

conda install mayavi

Mayavi makes high quality manipulable volume plots. Create a file scalar_field.py with the content

from mayavi import mlab
import numpy as np

x, y, z = np.mgrid[-10:10:20j, -10:10:20j, -10:10:20j]
s = np.sin(x*y*z)/(x*y*z)

scf = mlab.pipeline.scalar_field(x,y,z,s)
mlab.pipeline.volume(scf)
mlab.show()

Test programs for Mayavi. Mayavi should not error with

ImportError: Could not import backend for traits

• Mayavi changelog
• Mayavi releases

There may be spurious errors during install such as:

ModuleNotFoundError: No module named ‘vtk’

but Mayavi usually works anyway.

While ArXiv is among the earliest and best known preprint archives, more focused archives can provide easier access to a targeted audience with good reputation. Here are a few I’ve come across relevant to geosciences:

• Journal papers, Technical articles: EarthArXiv
• Posters / Conference Papers: ESSOAr
• Thesis (Masters, PhD) archiving: Thesis Commons

Display a login message using a message of the day (MOTD) with this script under /etc/update-motd.d/

The login message you’ll get will include:

Last Boot..........: 2020-04-12 14:29:29
Memory.............: 822 MB (Available) / 1021 MB (Total)
Load Averages......: 0.03, 0.08, 0.10 (1, 5, 15 min)
Running Processes..: 12
Free Disk Space....: 81 GB of 253 GB on /

Python script for MOTD:

#!/usr/bin/env python3
import sys
import psutil
from datetime import datetime
import shutil
from pathlib import Path

lastboot = datetime.fromtimestamp(psutil.boot_time())
vmem = psutil.virtual_memory()
drv = Path('~').expanduser().anchor
du = shutil.disk_usage(drv)

print("Last Boot..........:", lastboot)
print(f"Memory.............: {vmem.available//1000000} MB (Available) /  {vmem.total//1000000} MB (Total)")
if sys.platform == "linux":
    print(f"Load Averages......: {psutil.getloadavg()} (1, 5, 15 min)")

print("Total Processes....:", len(psutil.pids()))
print(f"Free Disk Space....: {du.free//1000000000} GB of {du.total//1000000000} GB on {drv}")

Ubuntu uses Python script “/usr/bin/landscape-sysinfo” to print a similar MOTD on login.

An increasing number of systems have multiple CPUs, say four, six or eight but may have modest RAM of 1 or 2 GB. An example of this is the Raspberry Pi. Ninja job pools allow specifying a specific limit on number of CPU processes used for a CMake target. That is, unlike GNU Make where we have to choose one CPU limit for the entire project, with Ninja we can select CPU limits on a per-target basis. That’s one important benefit of Ninja for speeding up builds of medium to large projects, and why we see increasing adoption of Ninja in prominent projects including Google Chrome. This is another reason why we generally strongly encourage using Ninja with CMake.

Specifically, CMake + Ninja builds can limit CPU process count via target properties:

• JOB_POOL_COMPILE: limit number of compile processes
• JOB_POOL_LINK limit number of link processes

The global JOB_POOLS property defines the pools for the targets.

Upon experiencing build issues such as SIGKILL due to excessive memory usage, inspect the failed build step to see if it was a compile or link operation, to determine which to limit on a per-target basis.

Example

Suppose that 500 MB of RAM are needed to compile a target and we decide to ensure at least 1 GB of RAM is available to give some margin. Thus we constrain the number of CPU processes for that target based on CMake-detected available physical memory. The appropriate parameters for your project are determined by trial and error. If this method still is not reliable even with a single CPU process, then a possible solution is to cross-compile, that is to build the executable on a more capable system for this modest system.

CMakeLists.txt includes:

set_property(GLOBAL PROPERTY JOB_POOLS one_jobs=1 two_jobs=2)

cmake_host_system_information(RESULT _memfree QUERY AVAILABLE_PHYSICAL_MEMORY)

add_library(big big1.c big2.f90)
if(_memfree LESS 1000)
  set_property(TARGET big PROPERTY JOB_POOL_COMPILE one_jobs)
endif()

Related: tell CMake to use Ninja

The Ninja build executable for Visual Studio location can be determined from the Visual Studio terminal:

where ninja

The factory Visual Studio Ninja version may be too old for use with CMake Fortran projects. If needed, replace the Visual Studio Ninja executable with the latest Ninja version, perhaps with a soft link to the ninja.exe desired. Add user permission to create symbolic links.