Scientific Computing

NetCDF4 is a powerful scientific data format based on HDF5 standard. NetCDF4 is directly usable by ParaView including the CF (Climate and Format) metadata convention. We have created a FindNetCDF.cmake that efficiently finds NetCDF libraries for use in CMake.

Example:

Place FindNetCDF.cmake file under project cmake/ then from CMakeLists.txt:

list(APPEND CMAKE_MODULE_PATH ${CMAKE_CURRENT_SOURCE_DIR}/cmake)

find_package(NetCDF COMPONENTS Fortran REQUIRED)

# example target
add_executable(main main.f90)
target_link_libraries(main PRIVATE NetCDF::NetCDF_Fortran)

Using HDF5 from any language can be intimidating if directly using the low-level HDF5 API. HDF5 interfaces have sprung up for popular languages that Make HDF5 trivially easy to use, such as Python h5py. Now Fortran also has an easy, powerful object-oriented and functional HDF5 interface named h5fortran. h5fortran builds and connects to the project using CMake or Meson. h5fortran uses Fortran 2018 standard code and is tested across multiple compilers including GFortran and Intel oneAPI. The efficient h5fortran interface is polymorphic (rank/dimension and type-agnostic) from scalars through 7-D arrays. A companion library for NetCDF is nc4fortran.

Examples

H5fortran is simpler to use than even Matlab HDF5 interface.

Functional interface example:

use h5fortran

call h5write('foo.h5', '/x', x)

call h5read('bar.h5', '/y', y)

Object-oriented interface examples:

use h5fortran

type(hdf5_file) :: h5f

call h5f%open('test.h5', status='new')

call h5f%write('/value1', 123.)

call h5f%close()

Using NetCDF4 from any language can be intimidating if directly using the low-level NetCDF4 API. NetCDF4 interfaces have sprung up for popular languages that Make NetCDF4 trivially easy to use, such as Python netcdf4. Now Fortran also has an easy, powerful object-oriented NetCDF4 interface named nc4fortran. nc4fortran builds and connects to the project using CMake or Meson. nc4fortran uses object-oriented Fortran standard code and is tested across multiple compilers including GFortran and Intel oneAPI. The efficient nc4fortran interface is polymorphic (rank/dimension and type-agnostic) from scalars through 7-D arrays. A companion library for HDF5 is h5fortran.

use nc4fortran

type(netcdf_file) :: hf

call hf%open('test.nc', status='new')

call hf%write('/value1', 123.)

call hf%close()

Nc4Fortran is simpler to use than even Matlab NetCDF4 interface.

IPyParaView from NVIDIA brings the interactive local / remote 3-D visualization power of ParaView to Jupyter Notebook. IPyParaView works on Linux, macOS and Windows.

First install ParaView, which is free and open-source. ParaView does not require a discrete GPU. ParaView will generally detect an use a discrete GPU if present, or use the CPU otherwise. CPU-integrated GPUs like Intel Iris are also used by ParaView.

We suggest using Miniconda Python and making a separate Conda environment for IPyParaView. Make the Python release match ParaView. Then install packages:

conda install ipython jupyterlab traitlets

Setup IPyParaView:

git clone https://github.com/nvidia/ipyparaview
cd ipyparaview

• Windows: from PowerShell run ./build.ps1
• macOS / Linux: run ./build.sh

Further details and examples are noted in the IPyParaView example repository.

CMake’s scripting language has some legacy characteristics as do other 1990s code languages. Editors such as Visual Studio Code have add-on CMake linting capabilities via CMake Tools. Deeper introspection is available via the CMake command-line options for uninitialized variables.

Uninitialized variables

Uninitialized variables can be less troublesome in CMake than other languages, but it’s better to be explicit by setting a new variable. Check for CMake uninitialized variables by:

cmake --warn-uninitialized -B build

This warning option can have false positives, or at least what the developer might consider false positives. Often the warning can be stifled by doing like:

set(myopt)

or

set(myopt "")

By default, Pytest checks all files matching its template. Pytest can select matching patterns of test files and test functions with the pytest -k ... option. Set breakpoint in a file to interactively debug in Spyder or other IDE.

Put default Pytest options in project “pyproject.toml”:

[tool.pytest.ini_options]
addopts = "-ra -v"

For Pytest to find and execute test scripts:

• each function name must begin with test_ e.g. def test_my1(a, b):
• each test file name must begin with test_ e.g. test_io.py
• each test file should be under a subdirectory somewhere in the project named tests/

To run self-tests on a specific file from the command line:

pytest src/mypkg/tests/test_my.py

NOTE: Generally using __file__ is not robust for packages. Consider using conftest.py fixtures and / or tmp_path where filesystem interaction is required from Pytest.

HDF5 soft (symbolic) links work like in filesystems–convenient as HDF5 UI is like a filesystem in a file. An unlimited number of soft links can point to the same resource. Soft links can dangle (point to an missing location) if needed.

HDFView

The HDFView GUI can create softlinks as well. This is handy when the program code changes variable name and one wants to keep HDF5 files compatible with old and new code (old name and new name). To create an HDF5 soft link in HDFView:

1. right click target variable name
2. select New → Link → Soft Link
3. when done, use the “close” button to save these changes

Note: if HDFView User Options are set to “Read Only” the “New” right-click option will be grayed out.

The soft link appears as any other variable in the HDFView GUI.

Other HDF5 link types

HDFView and other HDF5 interfaces can also create hard links and external links. An unlimited number of hard links can point to the same resource like soft links, but hard links are not allowed to dangle.

External links create a soft link to a resource in another HDF5 file. External links can be useful where datasets are very large, but one can treat a collection of files as one gigantic file from the user perspective.

References

• h5py soft links
• HDF5 library can create soft links

Wget can recursively download data or web pages. This is a key Wget feature that curl does not have. While curl is a library with a command-line front end, Wget is a command-line tool. Since recursive download requires several Wget options,

wget --recursive -np -nc -nH --cut-dirs=4 --random-wait --wait 1 -e robots=off https://site.example/aaa/bbb/ccc/ddd/

This downloads the files to whatever directory you ran the command in. To use Wget to recursively download using FTP, simply change https:// to ftp:// using the FTP directory.

Wget recursive download options:

--recursive

download recursively (and place in recursive folders on your PC)

--recursive --level=1

recurse but --level=1 don’t go below specified directory

-Q 1g

total overall download --quota option, for example to stop downloading after 1 GB has been downloaded altogether

-np

Never get parent directories (sometimes a site will link upwards)

-nc

no clobber – don’t re-download files you already have

-nd

no directory structure on download (put all files in one directory commanded by -P)

-nH

don’t put vestigial site name directories on your PC

-A

only accept files matching globbed pattern

--cut-dirs=4

don’t put a vestigial hierarchy of directories above the desired directory on your PC. Set the number equal to the number of directories on server (here aaa/bbb/ccc/ddd is four)

-e robots=off

Many sites will block robots from consuming data. Here we override this setting telling Apache that we’re (somewhat) human.

--random-wait

To avoid excessive download requests (that can get you auto-banned from downloading) we politely wait in-between file downloads

--wait 1

making the random wait time average to about 1 second before starting to download the next file. This helps avoid anti-leeching measures.

Using Pip is generally not recommended to install Spyder IDE because of the large number of dependencies. Get the latest stable Spyder release by:

conda update spyder

Users desiring the beta Spyder development version should consider the Spyder install procedure. Spyder official releases are usually available in Conda within a couple days.

Raspberry Pi can use system Spyder3:

apt install spyder3

Spyder shows up under “Programming” category of Start menu.

Matlab can directly create 3-D interactive plots of the Earth. An alternative is to export KML and use Google Earth to plot 3-D data with rendering of the Earth’s surface.

The plotting backend is WebGL, which is also used in uifigure. Matlab WebGL uses the GPU, including non-discrete GPUs built into the CPU as used in basic laptop PCs.

geoplot3 and geoglobe can generate a wide variety of intuitive, appealing and animated 3D plots.

fg = uifigure;
geoglobe(fg)

In Python, plotting data over a 3-D rendered Earth surface requires additional libraries like Mayavi or IPyParaView that serve as a frontend to VTK.