Scientific Computing

Git can use multiple transport protocols. We focus here on diagnosing connection issues for two commonly used Git transfer protocols: SSH and HTTPS.

Regardless of protocol used, Git trace capability can give more details. For example, before the problem commands, type:

• Windows: set GIT_TRACE=1
• macOS / Linux: export GIT_TRACE=1

Git clone over HTTPS can hang for a minute or more. Git clone from Overleaf will hang while other collaborators are making edits via the Overleaf web browser editor. Overleaf uses Git over HTTPS, and when Overleaf is busy (say, near federal grant deadlines) Git operations can be significantly slower on Overleaf. The verbose options of Git don’t add much information for Git over HTTPS connections, just a single additional line is added by --verbose like:

POST git-upload-pack (150 bytes)

Git over HTTPS can have a timeout:

git config --global http.lowSpeedLimit 1000
git config --global http.lowSpeedTime 60

This means that connections of less than 1 kB / sec for 60 seconds will be dropped. This avoids connection hangs, but could cause marginal connections to fail, so adjust parameters based on your connection and server if needed.

Git over SSH verbosity can be configured on a per-Git-command basis starting with:

git -c core.sshCommand="ssh -v"

This is quite verbose, so we usually wouldn’t add this to the Git config file.

Git over SSH timeouts can be addressed via SSH settings.

reference: GitLab Git troubleshooting connection tips

Saving PDF of plots from Matplotlib is an effective way to use Matplotlib plots in LaTeX or for general sharing of high quality plots. This can be accomplished by figure.savefig() to save to PDF from Matplotlib Simply use a filename with “.pdf” suffix.

Headless (fast) PDF generation: to save figures to disk without displaying them onscreen first, which is generally significantly faster:

from matplotlib.figure import Figure

fg = Figure(constrained_layout=True)
ax = fg.gca()
ax.plot(range(10))

fg.savefig('line.pdf', bbox_inches='tight', dpi=300)

LaTeX wrapfig.sty \wrapfigure is a great way to save space, but has several quirks to workaround. Overleaf also notes the touchiness of wrapfigure. The typical failure mode is that the wrapped figure will not appear, and the document text is wrapped for a few pages with blank column. The LaTeX log doesn’t give an error, just a warning about a missing figure reference.

Workarounds:

1. Don’t have wrapfigure next to a list (enumerated or itemized).
2. put a blank line before and after wrapfigure stanza in the document body

Example: for #2, in the .tex file have like

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Curabitur ut faucibus ex.

\begin{wrapfigure}{R}{0.5\textwidth}
\includegraphics{...}
...
\end{wrapfigure}

Donec sed commodo eros. Sed vehicula elit ac tincidunt lacinia.

A bug I fixed for Numpy 1.19.0 is Numpy internally did not correctly compare two-digit version numbers. To use GCC ≥ 10 with Numpy upgrade Numpy to at least version 1.19.0.

The PDF version output by LaTeX is set by \pdfminorversion and this also sets the newest PDF version that LaTeX can use via \includegraphics. If \pdfminorversion default or setting is too low, by default LaTeX only issues a warning like:

PDF inclusion: found PDF version <1.7>, but at most version <1.5> allowed

and the PDF-input graphic will be missing. The too-old default PDF version of LaTeX is increasingly becoming an issue as PDF creation tools are rightfully increasing their default PDF output version from the ca. 2003 PDF v1.5 specification.

To mitigate these issues, we do two things near the top of each project’s main LaTeX file:

1. create an error if the current PDF version is too old

   \pdfinclusionerrorlevel=1

2. set the default PDF version to ca. 2008 PDF v1.7

   \pdfminorversion=7

It’s OK for these PDF configuration lines to come after LaTeX directives.

3-D mesh wiregrid: minimal working example for the current version of Matplotlib. Pan, rotate, etc. in the figure window that opens.

from matplotlib.pyplot import figure,show
import numpy as np

x,y = np.meshgrid(np.linspace(0,2*np.pi), np.linspace(0,2*np.pi))

z = np.sin(x+0.5*y)
ax = figure().gca(projection='3d')
ax.plot_wireframe(x,y,z)
show()

Windows users may not have permission to create a symbolic link by default. Symptoms from Python may result:

from pathlib import Path

Path('x').symlink_to('y')

OSError: symbolic link privilege not held

Fix by enabling Windows symbolic links by enabling Windows developer mode.

Restart computer, and once again try Python script above to see that Python can create symbolic links.

Group Policy: press Windows r key (run dialog) and type

gpedit.msc

If gpedit.msc is not available, use Windows in Developer Mode as above.

Navigate to create user permission to create symbolic links:

Computer Configuration → Windows Settings → Security Settings → Local Policies → User Rights Assignment → Create symbolic links

type the user name and click “Check Names” then OK.

Reboot the computer (or logoff, logon) to make the permissions take effect.

Notes:

• Python os.symlink() on Windows
• symbolic links on Windows

When files need to be shared between two computers, whether at office, home or conference, major operating systems have peer-to-peer file sharing built in.

Windows

Windows Nearby sharing uses Bluetooth only, which means file transfer speeds are a tiny fraction (far slower) than even 3G tethered internet. Windows Nearby has the advantage like other operating systems of not requiring internet, but the real-world file transfer throughput is on the order of 1 Mbps, due to the limitations of Bluetooth itself and interference from other 2.4 GHz devices, even when the two laptops are less than 3 meters apart.

Windows Nearby file sharing is generally not practical for files over about 10 megabytes, due to the transfer rate of about 100..200 kilobytes / second in real-world use. That is, a 10 megabyte file takes about 2 minutes to transfer using Windows Nearby.

Apple

Apple AirDrop is much faster than Windows Nearby since AirDrop uses Bluetooth to negotiate a peer-to-peer WiFi connection that doesn’t rely on WiFi AP (router). Depending on the RF congestion and the nature of files transferred (few large files faster than many small files) the data throughput of AirDrop can be in the 100..500 Mbps range, 100 times faster than Windows Nearby sharing. It would be very beneficial if Windows Nearby would use adhoc WiFi like AirDrop.

Android

The deprecated Android Beam used NFC to negotiate a Bluetooth connection, which was very slow like Windows Nearby. Android Nearby is reported to a possible feature in Android 11, ChromeOS and Chrome browsers. We will hope that Android Nearby will use WiFi like AirDrop for useful speed.

Linkchecker-markdown is a very fast Python module that checks internal and external website links using the raw Markdown before rendering by the static site generator. Avoid broken internal links by using a Git pre-commit hook to check all internal links upon each “git commit”.

In the top directory of the Markdown website repo, do:

git config core.hooksPath .git/hooks

add the file .git/hooks/pre-commit (no .py extension)

Related: global Git pre-commit for checking Python PEP8, etc.

Continuous integration (CI) systems generally rely on an integer return code to detect success (== 0) or failure (!= 0). The error() function of Matlab / GNU Octave returns non-zero status that works well with CI systems. To specifically keep compatibility for a customer that needs outdated Matlab, we run Matlab CI tests using this script pair in the next section.

Matlab ≥ R2019a should use the -batch option to run from the command line instead of -r.

matlab -batch "assertSuccess(runtests)"

matlab_runner.m

This script is called by matlab_runner.py

function matlab_runner()
% for non-interactive use only (from system terminal)
% avoids command line quote escape issues
% fprintf() and exit() to be compatible with Matlab < R2019a

r = runtests;

if isempty(r)
  fprintf(2, 'no tests were discovered\n')
  exit(1)
end

if any(cell2mat({r.Failed}))
  fprintf(2, 'Failed with Matlab %s\n', version)
  exit(1)
end

exit(0)

end

matlab_runner.py

This calls matlab_runner.m. We use Python since it manages the command line much better than Matlab. Edit the variable “wanted_matlab” to test the required Matlab versions.

The exact method for switching Matlab versions may be different on your CI system.

#!/usr/bin/env python3
"""
Tests several versions of Matlab using Lmod module (for Linux)
relies on tests/version_runner.m
"module" requires shell=True
"""
import subprocess
import sys
import platform

if platform.system() != "Linux":
    raise SystemExit("This script for Linux only")

# the tests take several minutes, so we didn't test every possible version
wanted_matlab = ['2019b', '2022b']

failed = 0

for w in wanted_matlab:
    k = f"matlab/{w}"
    ret = subprocess.run(f"module avail {k}", stderr=subprocess.PIPE, text=True, shell=True)
    if k not in ret.stderr:
        print(f"SKIP: {k} not available", file=sys.stderr)
        continue

    mod_cmd = f"module load {k}"
    bat = "matlab -batch"

    ret = subprocess.run(mod_cmd + " && " + bat + " version_runner", text=True, shell=True, cwd='tests')
    if ret.returncode != 0:
        failed += 1


if failed == 0:
    print("OK:", wanted_matlab)
else:
    print(failed, " Matlab versions failed", file=sys.stderr)

Notes

Matlab -batch command line option makes error() return code 1. matlab -batch is so much more robust than matlab -r that users should generally switch to commands like:

matlab -batch test_myscript