Scientific Computing

3-D mesh wiregrid: minimal working example for the current version of Matplotlib. Pan, rotate, etc. in the figure window that opens.

from matplotlib.pyplot import figure,show
import numpy as np

x,y = np.meshgrid(np.linspace(0,2*np.pi), np.linspace(0,2*np.pi))

z = np.sin(x+0.5*y)
ax = figure().gca(projection='3d')
ax.plot_wireframe(x,y,z)
show()

Windows users may not have permission to create a symbolic link by default. Symptoms from Python may result:

from pathlib import Path

Path('x').symlink_to('y')

OSError: symbolic link privilege not held

Fix by enabling Windows symbolic links by enabling Windows developer mode.

Restart computer, and once again try Python script above to see that Python can create symbolic links.

Group Policy: press Windows r key (run dialog) and type

gpedit.msc

If gpedit.msc is not available, use Windows in Developer Mode as above.

Navigate to create user permission to create symbolic links:

Computer Configuration → Windows Settings → Security Settings → Local Policies → User Rights Assignment → Create symbolic links

type the user name and click “Check Names” then OK.

Reboot the computer (or logoff, logon) to make the permissions take effect.

Notes:

• Python os.symlink() on Windows
• symbolic links on Windows

When files need to be shared between two computers, whether at office, home or conference, major operating systems have peer-to-peer file sharing built in.

Windows

Windows Nearby sharing uses Bluetooth only, which means file transfer speeds are a tiny fraction (far slower) than even 3G tethered internet. Windows Nearby has the advantage like other operating systems of not requiring internet, but the real-world file transfer throughput is on the order of 1 Mbps, due to the limitations of Bluetooth itself and interference from other 2.4 GHz devices, even when the two laptops are less than 3 meters apart.

Windows Nearby file sharing is generally not practical for files over about 10 megabytes, due to the transfer rate of about 100..200 kilobytes / second in real-world use. That is, a 10 megabyte file takes about 2 minutes to transfer using Windows Nearby.

Apple

Apple AirDrop is much faster than Windows Nearby since AirDrop uses Bluetooth to negotiate a peer-to-peer WiFi connection that doesn’t rely on WiFi AP (router). Depending on the RF congestion and the nature of files transferred (few large files faster than many small files) the data throughput of AirDrop can be in the 100..500 Mbps range, 100 times faster than Windows Nearby sharing. It would be very beneficial if Windows Nearby would use adhoc WiFi like AirDrop.

Android

The deprecated Android Beam used NFC to negotiate a Bluetooth connection, which was very slow like Windows Nearby. Android Nearby is reported to a possible feature in Android 11, ChromeOS and Chrome browsers. We will hope that Android Nearby will use WiFi like AirDrop for useful speed.

Linkchecker-markdown is a very fast Python module that checks internal and external website links using the raw Markdown before rendering by the static site generator. Avoid broken internal links by using a Git pre-commit hook to check all internal links upon each “git commit”.

In the top directory of the Markdown website repo, do:

git config core.hooksPath .git/hooks

add the file .git/hooks/pre-commit (no .py extension)

Related: global Git pre-commit for checking Python PEP8, etc.

Continuous integration (CI) systems generally rely on an integer return code to detect success (== 0) or failure (!= 0). The error() function of Matlab / GNU Octave returns non-zero status that works well with CI systems. To specifically keep compatibility for a customer that needs outdated Matlab, we run Matlab CI tests using this script pair in the next section.

Modern Matlab tests instead of the “matlab_runner.m” below should use simply

matlab -batch "assertSuccess(runtests)"

matlab_runner.m

This script is called by matlab_runner.py

function matlab_runner()
% for non-interactive use only (from system terminal)
% avoids command line quote escape issues
% fprintf() and exit() to be compatible with Matlab < R2019a

r = runtests;

if isempty(r)
  fprintf(2, 'no tests were discovered\n')
  exit(1)
end

if any(cell2mat({r.Failed}))
  fprintf(2, 'Failed with Matlab %s\n', version)
  exit(1)
end

exit(0)

end

matlab_runner.py

This calls matlab_runner.m. We use Python since it manages the command line much better than Matlab. Edit the variable “wanted_matlab” to test the required Matlab versions.

The exact method for switching Matlab versions may be different on your CI system.

#!/usr/bin/env python3
"""
Tests several versions of Matlab using Lmod module (for Linux)
relies on tests/version_runner.m
"module" requires shell=True
"""
import subprocess
import sys
import platform

if platform.system() != "Linux":
    raise SystemExit("This script for Linux only")

# the tests take several minutes, so we didn't test every possible version
wanted_matlab = ['2019b', '2022b']

failed = 0

for w in wanted_matlab:
    k = f"matlab/{w}"
    ret = subprocess.run(f"module avail {k}", stderr=subprocess.PIPE, text=True, shell=True)
    if k not in ret.stderr:
        print(f"SKIP: {k} not available", file=sys.stderr)
        continue

    mod_cmd = f"module load {k}"
    bat = "matlab -batch"

    ret = subprocess.run(mod_cmd + " && " + bat + " version_runner", text=True, shell=True, cwd='tests')
    if ret.returncode != 0:
        failed += 1


if failed == 0:
    print("OK:", wanted_matlab)
else:
    print(failed, " Matlab versions failed", file=sys.stderr)

Notes

Matlab -batch command line option makes error() return code 1. matlab -batch is so much more robust than matlab -r that users should generally switch to commands like:

matlab -batch test_myscript

GCC / Gfortran 10 brought new warnings for arrays too big for the current stack settings, that may cause unexpected behavior. The warnings are triggered like:

real :: big2(1000,1000)

Warning: Array ‘big2’ at (1) is larger than limit set by ‘-fmax-stack-var-size=’, moved from stack to static storage. This makes the procedure unsafe when called recursively, or concurrently from multiple threads. Consider using ‘-frecursive’, or increase the ‘-fmax-stack-var-size=’ limit, or change the code to use an ALLOCATABLE array. [-Wsurprising]

This is generally a true warning when one has assigned arrays as above too large for the stack. Simply making the procedure recursive may lead to segfaults.

Correct the example above like:

real, allocatable :: big2(:,:)

allocate(big2(1000,1000))

For multiple arrays of the same shape do like:

integer :: M=1000,N=2000,P=500

real, allocatable, dimension(:,:,:) :: w,x, y, z

allocate(w(M,N,P))
allocate(x,y,z, mold=x)

As with the Intel oneAPI heap-arrays command-line options, there could be a penalty in speed by having large arrays drive off the stack into heap memory.

Newer version of compilers generally have more useful and detailed warning messages. As with any compiler, newer versions of Gfortran may require rebuilding other libraries linked with the Fortran compiler if the ABI presented by libgfortran changes. On Linux, one can switch Gfortran versions with update-alternatives. If experiencing errors getting any version of gfortran installed in Ubuntu, try:

add-apt-repository universe

The latest GCC / Gfortran for Ubuntu is available from the Ubuntu-test PPA. Add Ubuntu-test PPA by:

add-apt-repository ppa:ubuntu-toolchain-r/test

apt update

Install the most recent Gfortran listed at the PPA. Switch between compiler versions with update-alternatives.

Related:

• Windows: Install latest Gfortran
• macOS: get latest gfortran by brew install gcc

Astrometry.net is easy to use on Linux, macOS, and Windows. Windows uses Windows Subsytem for Linux for Astrometry.net. To get star index files, use downloadIndex.py.

Download/install the pre-compiled binary code:

• Linux / Windows Subsystem for Linux: apt install astrometry.net
• macOS Homebrew

The major steps in achieving a useful WCS starfield polynomial fit are:

1. source extraction (identifying star position in image frame)

2. quad asterism hashing, including tolerance for star position wiggling due to noise.

3. Match hash to star catalog, at least 3 choices are available:

   • USNO-B
   • Tycho-2
   • 2MASS (infrared)

4. Bayesian decision process, find extremely high probability solution or reject.

Astrometry.new tips and techniques

Optional compile: normally not necessary.

Prereqs:

apt install libcairo2-dev libnetpbm10-dev netpbm libpng12-dev libjpeg-dev zlib1g-dev swig libcfitsio-dev

curl -O https://astrometry.net/downloads/astrometry.net-latest.tar.gz

tar xf astrometry.net-*.gz

cd astrometry.net-*

make
make py
make extra

make install INSTALL_DIR=~/astrometry.net

Add to ~/.profile

export PATH="$PATH:$HOME/astrometry.net/bin"

do not use ~ to avoid error:

cannot find executable astrometry-engine

Uncomment inparallel in ~/astrometry.net/etc/astrometry.cfg (or /etc/astrometry.cfg)

Copy the star index files with downloadIndex.py

python downloadIndex.py

If it can’t find the index file, be sure ~/astrometry.net/etc/astrometry.cfg contains:

add_path /home/username/astrometry/data

~ or $HOME will NOT work!

Reference paper

Program giving azimuth/elevation for each pixel of sky image

Alternative: online astrometry.net image scaling

Eavesdropping / injection vulnerabilities allow unencrypted wireless mouse connection to be used as a keyboard by attackers to inject unwanted keystrokes, possibly taking over your PC. Force pairing allows unauthorized input to the PC. Logitech device firmware has distinct per-OS update procedures.

On Windows, the Unifying software is used to update firmware and pair receivers with mice and keyboards. In Logitech Unifying software click Advanced → Update Firmware

On Linux, fwupd supports updating Logitech Unifying receivers. Modern Linux distros will raise a prompt to seamlessly update Logitech receiver firmware.

On Linux, check firmware version and pair devices to the Logitech Unifying receiver with Solaar.

Fwupd: list all recognized devices, including firmware versions where applicable:

fwupdmgr get-devices

Continuous integration for program that plot or need a display can be tricky, since in many cases the CI doesn’t have an X11 display server. Workarounds include generating plots using X server virtual framebuffer (Xvfb) dummy X11 display server. This maintains code coverage and may allow dumping plots to disk for further checks

GitHub Actions: “.github/workflows/ci.yml”: assuming the project uses PyTest, the xvfb-action enables Xvfb for that command:

- name: Run headless test
  uses: GabrielBB/xvfb-action
  with:
    run: pytest

Related: Detect CI via environment variable](/ci-detect-environment-variable)