Ninja build on CentOS 7

The binary executables for Ninja 1.9.0 do not work on CentOS 7, the error is like:

$ ninja

ninja: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by ninja)
ninja: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by ninja)

This is a known issue with the Ninja release artifact build process, which is planned to be fixed for Ninja 1.10.0.

Workaround

Until Ninja 1.10.0 is released, use Ninja 1.8.2 on CentOS. This may work for other “older” Linux distros with similar errors.

Platform independent builds with Cmake

Related:


Note: we have moved most of our projects to Meson, which is generally easier, faster and robust.

A wide variety of programming languages are used by engineers and scientists. You can tie them all together (C, C++, C#, Cuda, Fortran, etc.) in a platform-independent and simple way using Cmake. These programs generate Makefiles for the program. Some choose to use a manually-created Makefile alone, but I prefer the platform-independent nature of Makefile generators such as Cmake.

Basic CMake example

Let’s take a single-file C++ program, like a science/engineering researcher might use. You will probably be linking the Math library at least, and Boost for flexible command-line input of numerous parameters. You might like to turn on additional compiler warnings to help avoid common coding pitfalls. Finally, you want to allow your colleagues on a wide variety of computers and operating systems to compile easily.

To do all of the above easily, Cmake has a simple syntax. Let’s consider an example CMakeLists.txt for a C++ and Fortran project, line by line.

Minimum version

cmake_minimum_required(VERSION 3.3)

Cmake 3.3 is new enough for many small projects, with features such as IN_LIST.

Language(s) selection

project(zakharov CXX)

Naming your project can help auto-install if you decide to someday. CXX is required to enable the hooks for the language(s) you used. The most frequently used include

taglanguage
CC
C#C#
CXXC++
CUDACUDA
FortranFortran

Languages that aren’t built into Cmake such as Pascal can be added via custom Cmake modules.

compiler options

if(CMAKE_Fortran_COMPILER_ID STREQUAL GNU)
  add_compile_options(-mtune=native -Wall -Wextra -Wpedantic -fexceptions -Warray-bounds)
endif()
-mtune=native
use specialized optimizations for your particular CPU
-Wall -Wextra -Wpedantic -Warray-bounds
turn on warnings for common programming mistakes
-fexceptions

more detailed debug info with no speed penalty–enabled by default on Clang.

set(CMAKE_CXX_STANDARD 11)
set(CMAKE_CXX_STANDARD_REQUIRED ON)

This project requires C++11 features, so if someone has an extremely old compiler not supporting C++11 they’ll get an error.

find_package(Boost REQUIRED COMPONENTS filesystem program_options)

The last two Boost lines are boiler-plate. The first line reflects that I use

filesystem
directory manipulation
program-options

advanced command-line parsing

add_executable(zakh zakh.cpp)
target_link_libraries(zakh PRIVATE Boost::filesystem Boost::program_options)
zakh

the exe file that will be created on compile, run with ./zakh.

zakh.cpp

the files making up “zakh”

Compiling a simple project with Cmake

It’s convenient to create a separate directory, typically build/ under your main code directory. Let’s say your main code directory is ~/code/zakharov, then do

# configure
cmake -B build/`

# build program
cmake --build build/ --parallel

# run program
./zakh

Let’s say you edit the code–rebuild and run by:

cmake --build build/ --parallel

./zakh

Normally you do not need to reconfigure CMake if just editing source code.

Notes

CMake alternatives include:

  • Meson (faster than CMake and recommended for new / updated projects)
  • SCONS (very slow builds, not recommended)
  • Autotools (slow builds, harder to implement, not recommended)

Linking HDF5 with CMake for C and Fortran

CMake links HDF5 into your C, C++, or Fortran program with just a few lines in your CMake file. An example CMake for writing network data to HDF5 in C: CMakeLists.txt.

A simple HDF5 read/write example is given below.

HDF5 prereqs

  • Linux: apt install libhdf5-dev
  • Mac: brew install hdf5
  • Windows MSYS2: pacman -S mingw64/mingw-w64-x86_64-hdf5

Windows Subsystem for Linux

apt install libhdf5-dev

and be sure /etc/wsl.conf includes:

[interop]
enabled=false
appendWindowsPath=false

CMakeLists.txt HDF5

Here’s two examples, one for C, one for Fortran. You can of course combine them.

Note: if your terminal has the Conda environment loaded and you keep getting the Conda HDF5 library, do first:

conda deactivate

before running the CMake configure commadn.

C HDF5 CMakeLists.txt

cmake_minimum_required (VERSION 3.10)
project(myproj C)

find_package(HDF5 REQUIRED COMPONENTS C)

add_executable(myprog myprog.c)
target_include_directories(myprog PRIVATE ${HDF5_INCLUDE_DIRS})
target_link_libraries(myprog PRIVATE ${HDF5_C_LIBRARIES})

Fortran HDF5 CMakeLists.txt

cmake_minimum_required (VERSION 3.10)
project(myproj Fortran)

find_package(HDF5 REQUIRED COMPONENTS Fortran Fortran_HL)

add_executable(myprog myprog.f90)
target_include_directories(myprog PRIVATE ${HDF5_INCLUDE_DIRS})
target_link_libraries(myprog PRIVATE ${HDF5_Fortran_LIBRARIES} ${HDF5_Fortran_HL_LIBRARIES})

HDF5 C example

The HDF5 syntax is quite similar (and simpler) for Fortran. See the hdf5 directory of Fortran 2018 examples for more.

#include "hdf5.h"
#define FILE "dset.h5"

int main() {

   hid_t       file_id, dataset_id,dataspace_id; /* identifiers */
   herr_t      status;
   int         i, j, dset_data[4][6], read_data[4][6];
  hsize_t     dims[2];

   /* Initialize the dataset. */
   for (i = 0; i < 4; i++)
      for (j = 0; j < 6; j++)
         dset_data[i][j] = i * 6 + j + 1;

   /* Create a new file using default properties. */
   file_id = H5Fcreate(FILE, H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT);

   /* Create the data space for the dataset. */
   dims[0] = 4;
   dims[1] = 6;
   dataspace_id = H5Screate_simple(2, dims, NULL);

   /* Create the dataset. */
   dataset_id = H5Dcreate2(file_id, "/dset", H5T_STD_I32BE, dataspace_id,
                          H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT);

   /* Write the dataset. */
   status = H5Dwrite(dataset_id, H5T_NATIVE_INT, H5S_ALL, H5S_ALL, H5P_DEFAULT,
                     dset_data);

   /* End access to the dataset and release resources used by it. */
   status = H5Dclose(dataset_id);

//------------------------------------------------------

   /* Open an existing dataset. */
   dataset_id = H5Dopen2(file_id, "/dset", H5P_DEFAULT);


   status = H5Dread(dataset_id, H5T_NATIVE_INT, H5S_ALL, H5S_ALL, H5P_DEFAULT,
                    read_data);

   for (i = 0; i < 4; i++)
      for (j = 0; j < 6; j++)
        printf("%d ",read_data[i][j]); // 1-24

   /* Close the dataset. */
   status = H5Dclose(dataset_id);

   /* Close the file. */
   status = H5Fclose(file_id);
}

HDF5 compiler macros

As an alternative (or companion to) CMake, one can use compiler macros for HDF5. HDF group provides compiler macro h5cc linking the needed HDF5 libraries upon installing libhdf5-dev:

h5cc myprog.c func.c -lm
  • h5cc: C
  • h5c++: C++
  • h5fc: Fortran

Best format for shared USB drives and SD cards

EXFAT is the most convenient format for sharing files between MacOS / Linux / Windows / Android. ExFAT does not support symbolic / soft links.

Operating systems have supported EXFAT for many years, since:

  • Windows: XP
  • MacOS: 10.6.3
  • Linux: CentOS 5, Ubuntu 10.04, etc.
  • Android: 4

On Linux (including Ubuntu 18.04), install ExFAT support:

apt install exfat-utils

Exceptions

Use FAT32 instead of EXFAT for:

  • Test equipment (e.g. Agilent)
  • some older USB drive accepting printers (print direct from USB stick)

Other file systems

These file systems are commonly, but not universally supported across OS’s.

NTFS

  • MacOS (including MacOS 10.15 Catalina): NTFS is read-only by default.
  • Android needs additional third-party software for NTFS

FAT32

FAT32 is readable by Windows / MacOS / Linux / Android, but is limited to 4 GB file size. With many USB HDDs over 2 TB, you would have to break up your HDD into multiple partitions for FAT32 – quite inconvenient. You could use FAT32 for a small USB flash drive.

Note: For old printers “USB Direct Print” it may be that only single partition FAT32 is supported. Keep a small USB drive formatted to FAT32 for this purpose.

Summary

ExFAT is generally the best choice for universal support on USB HDD. For USB flash thumb drives and SD cards, I format most to ExFAT. Keep a small flash drive formatted to FAT32 for the USB port on old USB direct print printers.

MacOS hardware requirements vs. version

Those who need to keep MacOS-capable hardware around for development or maintenance purposes, or simply as a form of conserving resources may be interested to know how Apple is increasing MacOS requirements over time. In general, if the Apple computer is too old for a currently supported MacOS version, consider installing Linux, which normally works well on Apple computer hardware. Apple maintains a list of no longer supported Apple hardware.

MacOS hardware requirement vs version

MacOS requirements by version.

The red dotted line depicts the oldest supported version of MacOS.

Data source is Wikipedia.

Using CMake on Windows

CMake on Windows is installed via the cmake*-win64-x64.msi graphical installer. Do not use sudo or Run As Administrator for cmake in general.

Compile programs using CMake

  1. Navigate to the directory containing the file CMakeLists.txt using the Windows Terminal / Command Prompt.
  2. configure the build. This is normally run only once unless making major project changes.

    cmake -G "MinGW Makefiles" -DCMAKE_SH="CMAKE_SH-NOTFOUND" -B build .
  3. compile the program. This is the command run each time you make a change to the project code.

    cmake --build .
  4. Optionally, install the program with

    cmake --build . --target install

Generator selection

On Windows, CMake defaults to Visual Studio and Nmake. The cmake options above direct the build to MinGW. If you wish to make this change permanent for CMake ≥ 3.15, set the Windows environment variable CMAKE_GENERATOR to “MinGW Makefiles”. This can still be overridden if needed like

cmake -G "Visual Studio 16 2019"

sh.exe error with cmake

The nuisance error from cmake about sh.exe being on the Path. This error also happens with CMake Windows builds on Azure Pipelines and GitHub Actions.

sh.exe was found in your PATH, here: C:/Program Files/Git/user/bin/sh.exe For MinGW make to work correctly sh.exe must NOT be in your path. Run cmake from a shill that does not have sh.exe in your PATH. If you want to use a UNIX shell, then use MSYS Makefile

Eliminate this message by adding -DCMAKE_SH="CMAKE_SH-NOTFOUND" to the cmake command, like:

cmake -G "MinGW Makefiles" -DCMAKE_SH="CMAKE_SH-NOTFOUND" ..

Notes

Use ** instead of pow in Python

In Python, x**y is much faster than:

Julia is more than 5 times faster than Python at scalar exponentiation, while Go was in-between Python and Julia in performance.

Python

Benchmarking was the same for integer or float base or exponent.

Python testing done with:

  • Python 3.7.4
  • Ipython 7.8.0
  • Numpy 1.16.5

** operator

The ** operator in Python also has the advantage of returning int if inputs are int and arithmetic result is integer.

10**(-3)
8.22 ns ± 0.0182 ns per loop (mean ± std. dev. of 7 runs, 100000000 loops each)

pow(10, -3)
227 ns ± 0.313 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

math.pow(10, -3)
252 ns ± 1.56 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

numpy.power(10., -3)
1.5 µs ± 2.91 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

Numpy is known in general to be slower at scalar operations than Python-native operators and Python built-in math. But of course Numpy is generally a lot faster and easier for N-dimensional array operations.

Julia

Julia 1.2.0 was likewise benchmarked for reference on the same computer.

First we installed Julia BenchmarkTools:

import Pkg
Pkg.add("BenchmarkTools")

The Julia wallclock time for exponentiation was the same for float and int as with Python.

3.399 nanoseconds

Go

Go 1.13.1 was benchmarked:

go test -bench=Power
BenchmarkPower-12       33883672                31.8 ns/op

go benchmark reference

Python flatten list of lists into list

For scenarios where a function outputs a list, and that function is in a for loop or asyncio event loop, the final output will be a list of lists, like:

x = [[1,2,3], [4, 5, 6]]

This may be inconvenient for applications where a flattened list is required. The simplest and fastest way to flatten a list of lists in like:

import itertools

x = [[1,2,3], [4, 5, 6]]

x_flat = list(itertools.chain(*x))

which results in

[1, 2, 3, 4, 5, 6]

GNSS data abbreviations

Some of the most fundamental GNSS measurements retrieved from GNSS receivers include:

  • CNR: carrier to noise ratio or C/N0. The estimation technique varies between receivers. Typical values in the range 30..50 [dB Hz]
  • PSR: Psuedorange [meters]
  • ADR: accumulated Doppler range–carrier phase measurements [cycles]

Using networks of GNSS receivers along with appropriate post-processing techniques, estimated maps of vertical TEC (integrated electron density) can be derived:

Haystack TEC map

Haystack TEC map https://www.haystack.mit.edu/eclipse.html

Reference

GitHub / GitLab Pages to Netlify

While both GitHub Pages and GitLab Pages are adequate for most personal, group and project pages, when website size and / or traffic have grown beyond what is feasible for these solutions, a more comprehensive hosting provider like Netlify may be considered. Netlify provides its own CDN, so those that had been using Cloudflare for DNS and CDN can configure Cloudflare to provide only DNS, if they so choose. Netlify is free for single users, allowing a private GitLab, GitHub or Bitbucket repo (or other suitable source) to deploy to a public custom domain HTTPS website. SSL certificates can be user-provided or can be created through Netlify for your custom domain (e.g. https://www.example.com).

Why transfer site to Netlify

Netlify provides a comparison of GitHub Pages and Netlify. GitLab Pages allows user choice of static site generator (Hugo, Jekyll, etc.), while GitHub Pages allows only Jekyll. GitLab Pages private repos have a runtime limit of 2000 minutes/month. Netlify allow 3 builds / minute and 100 GB / month on the free tier, with 300 build minutes/month. For sites that are becoming very popular, GitHub Pages will simply want you to move elsewhere, while Netlify will have a paid plan to offer. This process may be too burdensome for those with limited IT or bandwidth resources, or simply the lack to time to learn how to do this.

Netlify uses webhooks to detect a git push to the website GitLab repo, and then builds the site. Netlify has a CDN and DDoS protection built-in. Even if the other features aren’t needed, a key feature is the ability to have the website code in a private repo with unlimited public website deployments and traffic.

Build minute limits (such as on GitLab and Netlify) can legimately be worked around by building the site locally on your laptop and pushing the publish-ready HTML.

Transfer site to Netlify

Note: This process may take down your site for a day or two if things go wrong. Even under normal conditions, all site visitors may need to allow an HTTPS exception due to SSL certificate error since Netlify requires all DNS servers to update before generating the domain certificate.

  1. if not already on GitLab, copy your website repo to GitLab (any name repo is fine).
  2. disable Auto DevOps and ensure no file named .gitlab-ci.yml exists.
  3. Login to Netlify using Gitlab, which will ask for your website repo.
  4. pick a custom Netlify subdomain like mycompany.netlify.com. Ensure this site is totally working before proceeding.
  5. Set Cloudflare or whatever your DNS provider is to point CNAME or A to mycompany.netlify.com (THIS IS THE PART THAT CAN TAKE YOUR MAIN WEBSITE DOWN!)
  6. Under Netlify Domain Management → HTTPS → Verify DNS config, ensure the verification completes. Until the DNS change propagates worldwide, your main HTTPS domain visitors are getting SSL verification errors. They can use http://mycompany.com instead of https://mycompany.com temporarily. Do this at a low traffic time range! If you were using Cloudflare CDN, the old records may point to DigitalOcean while the new records point to *.netlify.com.
  7. Optionally, under Netlify → Build & deploy → Post Processing → Asset Optimization consider using these to improve website speed.