Scientific Computing

Goodnow Farms Chocolate

Modern scientific practices are concerned with obtaining results in a socially conscious and responsible manner, minimizing external impacts. Recent papers including Strubell, Ganesh, McCallum 2019 “Energy and Policy Considerations for Deep Learning in NLP arXiv note the energy consumption impacts of machine learning and AI. A tech ethics leader pushing for responsible AI was abruptly fired. Critical challenges in efficient food and packaging supply chains include plastic recycling, plastic ocean patches. and human factors such as child slavery in chocolate production. The American Geophysics Union (AGU) reorganized sections including Science and Society and GeoHealth oriented around the intersection science has with human well-being and government policies.

Companies increasingly use systems approaches to product quality and minimizing external costs. Goodnow Farms chocolate production of top-quality confectioneries considers the whole cacao lifecycle. Each chocolate type has an associated story and photo book showing the local farmer’s process.

.gitignore Meson subprojects

Meson projects using Meson subprojects have a project directory structure like:

meson.build
main.c
subprojects/
  lapack/
  lapack.wrap

where “subprojects/lapack/” is automatically created as a Meson subproject from the metadata in “subprojects/lapack.wrap”. It’s desirable to ignore the subprojects code directories in Git.

For Meson subprojects, using the negate .gitignore syntax Git will ignore subdirectories but track the subprojects/*.wrap files, by including in the project top-level .gitignore:

/subprojects/*

!/subprojects/*.wrap

This technique can be applied to similar non-Meson scenarios.

NOTE: The .gitignore syntax has a precise meaning. For example, /subprojects/* means to ignore the directory contents of top-level directory “subprojects”, but not the directory itself. This is important for the next stanza !/subprojects/*.wrap, which means to not ignore files with .wrap suffix in top-level directory “subprojects”.

Thanks to @lunasorcery for pointing out the correct .gitignore syntax.

Create empty file

Empty files can be used as a mean to pass information between programs in a persistent manner, or a form of configuration. Empty files can be used on web servers to declutter 404 logs of endless bot scans.

For the examples below, we assume relative filename “path/to/empty”, where directories “path/to” already exist. We assume that a file does NOT exist at the filename, and it may be overwritten if it exists.

NOTE: For simplicity, some of these examples overwrite existing files, but some do not overwrite. Extra code is required to give a consistent behavior. All examples at least create an empty file if one doesn’t exist.

Windows users often use PowerShell. Create empty file with PowerShell (on any operating system):

New-Item path/to/empty

For Linux, macOS and other Unix-like systems, most shells create empty file like:

touch path/to/empty

Python creates an empty file by:

pathlib.Path('path/to/empty').touch()

Fortran creates an empty file by:

open(newunit=u, file='path/to/empty')
close(u)

Create an empty file in C++:

#include <fstream>

std::ofstream output("path/to/empty");

Netlify redirects

Those running a public web server are frequently subject to bot scans. While using best practices is the proper approach, there may still be a lot of 404 log results from bots. Declutter the log by redirecting bot URLs to a single zero-byte empty page while still returning code 404.

Another choice is to return HTTP status codes. Netlify only allows a few _redirects HTTP status codes. We redirect bot URLs to /bot.html and return a 404 code using an empty file.

Empty files: since these files are legitimate, but may not be used by your site, we create empty files for these:

favicon.ico
ads.txt
app-ads.txt

The Netlify _redirects file should be in the site publish directory. For Hugo it may be “public/_redirects”. We prefer the _redirects file to netlify.toml, because the redirect syntax is much more concise in _redirects files. Use Netlify Playground to instantly test file syntax.

A _redirects file for a Hugo blog may need editing to be suitable for your site.

References:

Pixel 5 improvements over Pixel 4

The Pixel 4 and Pixel 5 have multi-frequency GPS L1 and L5 bands for substantially increased navigation accuracy, as well as increased usefulness in citizen science GNSS projects. Dual frequency GNSS receivers enable additional types of ionospheric measurements not readily available with single frequency L1 GNSS receivers.

The Google Pixel 4 has been noted as having a key flaw–battery life too often less than 12 hours. Even disabling features such as Project Soli / Motion Sense and having the latest Android 11 did not greatly increase battery life beyond 18 hours. Short battery life was historically also quite an issue across numerous iPhone models, but is no longer such an issue.

The Pixel 5 uses a high-midrange CPU instead of top-end CPU in Pixel 4–the Pixel 5 CPU seems nearly as speedy as the Pixel 4.

The display panel itself is used for the top speaker–this restricts fidelity of the speaker, making it greatly emphasize midrange frequencies. This is fine for voice, but if you wish to listen to stereo orchestra music, you’d benefit from headphones.

The Pixel 5 does not have the “squeeze for Google Assistant” or Motion Sense.

The baseline battery life in the Pixel 5 is about 24 hours, which can be pushed to 36+ hours with battery saver–much better than Pixel 4. All in all, even if a used Pixel 4 is substantially cheaper, go for a used (or new) Pixel 5 instead.

Reference: dual frequency GNSS phone model spreadsheet

Use PNG for favicon.ico instead

We don’t use favicon.ico because the .ico format is uncompressed ~ 16 kB. Instead we suggest creating a compressed favicon that works in place of favicon.ico for all web browsers we tried. These files are typically 1..2 kB instead of 16 kB, which is significant for a text-based website.

favicon.png
favicon-16x16.png
favicon-32x32.png

Then we create an empty file favicon.ico that web browsers will ignore, and use the PNG instead.

Honeypot URLs to declutter 404 log

Bots scan a set of URLs that can clutter the site 404 log. One can simply ignore these items, apply redirects, return HTTP codes, or create empty files. Creating a lot of empty files can be a bit awkward. Redirection is a server / service specific configuration that can possibly return HTTP error codes. We choose to redirect bots to an empty file.

Pytest.importorskip test skipping

Pytest allows clean handling of numerous continuous integration challenges. Many programs use advanced plotting that may not be desired to test on continuous integration. For example, one might opt not to install matplotlib as part of CI runs. We put plotting routines in a separate file to facilitate this, allowing the core module functions to have a bare minimum set of prereqs. This is useful for HPC, CI and cloud scenarios that either may not plot locally or that may use varied plotting systems depending on whether running on web or local PC.

pytest.importorskip() is a vital tool to handle to cleanly and clearly handle these scenarios.

Imagine a Python package file hierarchy like:

mymod/
  __init__.py
  plot.py
  adv.py
  tests/
    test_mod.py
    test_plots.py

Skipping function imports is demonstrated in this example. Suppose optional module mymod.adv() requires scikit-image, which itself has a lot of prereqs. One might opt to have CI not install scikit-image, and then use pytest.importorskip() to cleanly skip complicated, optional tests on CI, HPC etc.

import pytest
import mymod


def test_advanced():

    adv = pytest.importorskip('mymod.adv')

    ...

Skipping module imports is demonstrated in this example. Suppose “test_plots.py” has every function requiring Matplotlib. The whole file may be skipped instead of skipping each function like:

import pytest
try:
    import matplotlib
except ImportError:
    pytest.skip("Matplotlib not available", allow_module_level=True)


def ...

Conditional skipping can handle more complicated cases by making a decorator with pytest.skipif.

import pytest
import os

@pytest.mark.skipif(os.name=='nt', reason='only for Unix-like systems')
def test_unix():
    ...

If you have more than one function to skip with the same test skip condition, make your own decorator:

import pytest
import os

unixonly = pytest.mark.skipif(os.name=='nt', reason='only for Unix-like systems')


@unixonly
def test_unix_plot():
    ...


@unixonly
def test_unix_network():
    ...

Open Source Criticality Score

Google via the OSSF introduced a Python API to compute Criticality Score of open-source projects. While there will continue to debate about how exactly to clip and weight parameters, being able to reduce metrics to a single number gives the community a quick first pass to judge which projects need help. Like h-index for academic bibliographies, Criticality tells only part of the picture. What about the score of fundamental libraries that have a large number of unhandled pull requests and infrequent updates.