Scientific Computing

HDF5 Fortran dimension order

The Fortran array dimension order is column-major, meaning the first dimension changes fastest in memory. This is consistent with how Fortran and Matlab store arrays in memory. When indexing arrays, particularly in a loop, significant performance differences can arise depending on the dimension order. Accessing elements in the fastest-changing dimension first is generally more efficient as it’s memory contiguous, reducing cache misses and improving performance.

The HDF5 document on C versus Fortran interfaces explains how HDF5 handles arrays in different programming languages. Most languages, including C, C++, Java, and Python use row-major order, where the last dimension changes fastest in memory, which is also how the HDF5 interface to those languages works. For Fortran and Matlab, the HDF5 interface expects arrays in column-major order, consistent with Fortran’s memory layout. This column-major array ordering is also true for the HDF5 APIs of the h5fortran object-oriented Fortran HDF5 interface and the stdlib for Matlab libraries.

The HDF5 Library stores arrays along the fastest-changing dimension. This approach is often referred to as being “in C order.” C, C++, and Java work with arrays in row-major order. In other words, the row, or the last dimension, is the fastest-changing dimension. Fortran, on the other hand, handles arrays in column-major order making the column, or the first dimension, the fastest-changing dimension.

C++20 adoption by popular projects

C++20 standard (issued December 2020) is long supported by all modern C++ compilers. In particular Microsoft Visual C++ has been steadily adding support for C++20 (and newer standards).

Major and/or popular projects are already adopting C++20 features in their codebases. C++20 codified several features that had been compiler-specific extensions. Some examples:

One of the biggest discussions about C++20 is modules, which improve compilation times and provide better encapsulation of code compared to traditional header files. The C++ ecosystem from compiler to build system works for C++20 modules with CMake support, but is still maturing.

Migrating the build configuration of a project to use C++20 standard when available can reveal oversights and fragile assumptions in the codebase, particularly around features that have changed or been newly introduced in C++20. An example of this is the numerous bugfixes surrounding a default build in C++20 for CMake, despite CMake not yet requiring C++20 for its own build.

GitHub Gist branch must be "main"

GitHub Gist only recognizes the “main” branch. Git commits on any other branch will not be visible in the Gist interface.

Check the current branch with:

git branch --show-current

If commits were accidentally made in a non-main branch, change the name of the branch to “main” to make the commits visible in the Gist interface.

git branch -m main

After renaming the branch, it may be necessary to force push to update the remote Gist. First, ensure the remote content is in sync with the local branch to avoid overwriting any changes.

# --force overwrites the remote branch with local changes
git push origin main [--force]

Check if the old branch was pushed

git ls-remote origin

If the old non-main branch (perhaps “master”) still exists on the remote, consider deleting it to avoid confusion.

git push origin --delete master

Perhaps globally configure the computer to always use “main” as the default branch for new repositories:

git config --global init.defaultBranch main

Get HDF5 library version

Computing platforms may have multiple versions of the HDF5 library installed. The HDF5 library version currently being used can be determined by the program as in the following examples by language. The HDF5 library can write backward-compatible HDF5 files by HDF5 version bounds. Currently there is no way to introspect the HDF5 library version required to read a specific HDF5 file other than trying to read the HDF5 file.

Matlab HDF5 library version

Matlab release vs. HDF5 library version:

Matlab HDF5
R2024b (Update 3) 1.14.4.3
R2024a 1.10.11
R2023b 1.10.10
R2022a 1.10.8
R2021b 1.10.7
R2015a 1.8.12

Check Matlab HDF5 library version:

[major, minor, rel] = H5.get_libversion()

The HDF5 functions available in Matlab are listed by:

funcs = string(H5ML.get_function_names())

For example, in Matlab R2026b there are 363 HDF5 functions available out of over 800 HDF5 library functions.

Python HDF5 library version

Check Python h5py HDF5 library version:

import h5py

print(h5py.version.hdf5_version)
print(h5py.h5.get_libversion())

The Python h5py library provides an interface to HDF5 functions as listed.

C, C++, Fortran HDF5 library version

In C, C++, Fortran, check HDF5 library version with H5get_libversion().


Related: get NetCDF library version

Recovering a dropped Git Commit

After pushing a commit or merging a pull request, one may accidentally drop a Git commit from branch history. If the commit was part of a pull request, GitHub will not reopen a closed pull request whose source branch no longer contains the commit. Instead, check whether the commit still exists in git reflog by searching for its message or hash.

git reflog

If the commit exists in the reflog, recover it using the commit hash:

git checkout -b recovered-branch <commit-hash>

If the commit is not in the reflog, it may still exist as an unreachable object or in another branch or remote. Note the reflog is local: it only records commits that existed in the local repository.

To search for unreachable (dangling) commits:

git fsck --unreachable

For example, to check all branches for the specific commit:

git branch --contains <commit-hash>

To check all remotes for the commit:

git fetch --all
git branch -r --contains <commit-hash>

If the commit is found in another branch or remote, recover it by checking out that branch or creating a new branch from the commit hash.

If the commit is only found in someone else’s repo (perhaps the person that made the pull request), obtain the commit URL and apply it like:

curl -L https://github.invalid/user/repo/commit/hash.patch | git am

git am is used instead of git apply because it creates a commit that preserves the original author and commit message, ready to push.

Scotch and ParMETIS Packaging

Major Linux distros including Ubuntu 26.04 and Debian stable have adopted the Parmetis compatibility layer provided by Scotch for parallel graph partitioning and sparse matrix reordering. Therefore “libparmetis-dev” package has been dropped in favor of the free Scotch-provided compatibility layer in “libscotchparmetis-dev”, which provides libparmetis.so and the headers required for ParMETIS compatibility.

Git Bisect project with Git Submodules

The standard git bisect command helps quickly identify a specific commit that introduced a bug in a Git repository. In projects where branches with lots of commits were previously merged, there may be known-bad commits with build failure unrelated to the current bisect search. It’s possible to skip such commits during the bisect process, or avoid specific commits altogether.

For projects using Git submodules it’s useful to set option submodules.recurse true

git config --global submodule.recurse true

that works for most Git commands and avoids the need during “git bisect” to also have to “git submodule update –init –recursive” at each step of the bisect process.

git bisect start

# if you already know this current commit is good or bad, tell Git about it
git bisect <good|bad> <commit>

# test the current commit by running your test code / program / script
git bisect run <test command>

# ... repeat the above two steps until Git finds the first bad commit

# finish the bisect process and return to the original branch
git bisect reset

Troubleshooting

If running into errors about “dirty” submodules and/or lots of deleted files when Git bisecting or just switching Git branches, try disabling Git submodule.recurse for the command:

git -c submodule.recurse=false submodule update --init --force --recursive

The global submodule.recurse=true setting is convenient but makes checkouts fragile when any nested submodule gets into a bad state.

Lossless convert PNG image stack to PDF

The Python package img2pdf losslessly converts PNG images to PDF.

Install it with:

pip install img2pdf

The command line program “img2pdf” can do many tasks, but even more power comes from making a brief Python script. Here are simple command line examples.

Convert a single PNG to PDF:

img2pdf in.png -o out.pdf

Convert multiple PNG images to a single PDF:

img2pdf *.png -o joined.pdf

Rotate each page of a PDF from the command line

A common issue with document scanning is that pages may have been inserted in the wrong orientation, making the PDF difficult to read with upside-down text. This is trivially fixed with the QPDF command line tool.

qpdf --rotate=180 in.pdf out.pdf
in.pdf
input PDF file needing rotation out.pdf
output PDF file with rotated pages
--rotate=180
rotate each page 180 degrees

QPDF can do vastly more operations on PDF files as seen in the QPDF documentation. Poppler is oriented toward rendering and extracting content from PDF files rather than modifying them like QPDF, which does not render PDF files.

Install QPDF

  • Linux: apt install qpdf or similar
  • macOS: brew install qpdf
  • Windows: winget install QPDF.QPDF

On Windows, QPDF is installed under $Env:ProgramFiles/qpdf*/bin/qpdf.exe

Open same PDF multiple times

Opening the same PDF file multiple times is particularly useful when reviewing journal article drafts to view figures simultaneously with text describing the figure.

Okular is a free lightweight PDF viewer for Linux, macOS and Windows that trivially opens the same PDF multiple times by just selecting through the File → Open menu as many times as desired.

Okular is installed by:

  • Linux apt install okular or similar
  • macOS - see Okular download link above
  • Windows winget install KDE.Okular