Matlab package import like Python

Matlab users can package code projects as toolboxes and/or packages. The packages format works for ancient versions of Matlab, before even R2006a as well as GNU Octave. The toolbox format works for Matlab ≥ R2016a and not GNU Octave. The packages format brings benefits to toolboxes as well.

Matlab namespaces

A key issue with Matlab vs. Python arise from that Matlab users often add many paths for their project. If any function names clash, there can be unexpected behavior as it’s not immediately clear which function is being used without further investigation of path ordering. As in Python and other languages, there is considerable benefit for using a package format where the function names are specified in their namespace.

addpath example

To enable Matlab package format, we explain by example. Suppose a project directory structure is like:

myproj
  utils
    mem1.m
  conversion
    deg1.m
  sys
    disk1.m

To use these functions, the end users do:

addpath(genpath('myproj'))

This is where the namespace can have clashes, and with large projects it’s not clear where a function is without further introspection.

package example

To make this project a Matlab / Octave package, change the subdirectories containing .m files to start with a “+” plus symbol:

myproj
  +utils
    mem1.m
  +conversion
    deg1.m
  +sys
    disk1.m

Now the end users will simply:

addpath('myproj')

and then access specific functions like:

myproj.utils.mem1(arg1)

Then multiple subdirectories can have the same function name without clashing in the Matlab namespace. Support the function “mem1” is used frequently in another function. To avoid typing the fully resolved function name each time, use the import statement:

function myfunc()

import myproj.utils.mem1

mem1(arg1)

mem1(arg2)

Matlab toolbox .mltbx

Matlab .mltbx toolboxes became available in R2016a. The Matlab-proprietary toolbox format also allows end users to create their own packages containing code, examples and even graphical Apps. In effect .mltbx provides metadata and adds the package to the bottom of Matlab path upon installation. The installation directory is under (system specific)/MathWorks/MATLAB Add-Ons/Toolboxes/packageName. Whether or not the project uses .mltbx, the namespace of the project is kept cleaner by using a Matlab package layout.

Intel MPI on Windows

On Windows, the Intel C, C++ and Fortran compilers present Visual Studio-like command line options. The correct version of Visual Studio must be installed on Windows for Intel compilers to work. C and C++ use the icl compiler on Windows.

The free Intel oneAPI with HPC toolkit includes the Intel MPI library, which provides mpiexec needed to run MPI programs and MPI compiler wrappers.

Loading Intel compiler environment

Most users use the Intel oneAPI command prompt. Alternatively, run “compilervars.bat” script to enable the Intel compilers for each session. “psxevars.bat” is not appropriate for this setup. For convenience, make a batch script like %userprofile%\intel.bat containing:

"C:\Program Files (x86)\inteloneapi\compiler\latest\windows\bin\compilervars.bat" intel64

set FC=ifort
set CC=icl
set CXX=icl

Intel MPI on Windows is only for Intel compiler

Unlike for Linux Intel MPI, Windows Intel MPI is only for the Intel C, C++ and Fortran compilers and Visual Studio.

Notes

Although not often needed, a separate username can be used for Windows Intel MPI jobs by from Command Prompt:

runas /user:username cmd

Environment variables are not passed to the new window, so it may be necessary to run Intel compilervars.bat again. It’s possible to register the user credential in the Windows registry.

Major changes in Gfortran by version

Gfortran and Intel oneAPI are the most advanced modern Fortran compilers. Useful Fortran 2018 enhancements include:

oldest Gfortran version

Gfortran 8 is the oldest version currently maintained.

Intel oneAPI

  • Intel oneAPI has “full” Fortran 2018 support.

Gfortran major changes

Gfortran 10

  • select rank – making assumed rank arguments actually useful.

Gfortran 9

  • added random_init() to initialize the random generator seed…randomly

Gfortran 8.1

  • Optimization: automatic nested loop exchange with do concurrent
  • Checks: Actual argument array with too few elements for dummy argument now errors
  • Polymorphism: initial support for parameterized derived types (simply define kind at initialization)
  • Coarray: Initial support for teams

(Gfortran 8.2 and 8.3 did not introduce new Fortran features)

Gfortran 8

  • std=f2008ts deprecated, do not use as it errors on compiling error stop for Gfortran < 8.
  • std=f2018 added

Gfortran 7

  • Polymorphism: derived type IO select type etc. fully supported from Fortran 2003.
  • Fortran 2018: Non-constant stop and error stop codes
  • Compatibility: -fdec- options help compiling very old non-standard code that was previously only compatible with Intel oneAPI (of DEC heritage).

Gfortran 6

Fortran 2008: submodule support, useful for large projects to save compilation time and allow powerful use scenarios

Fortran 2003: improved support for deferred-length character. These are very useful for avoiding bothersome trim() everywhere. Example:

character(256) :: argv
character(:), allocatable :: filename

call get_command_argument(1, argv)
filename = trim(argv)

end program

In this example, filename is now exactly the length required to fit the text. If the filename (in argv) was “hello.txt” then len(filename) is 9. That is, len_trim == len with auto-allocated characters (unless subsequently rewritten with a shorter string).

Practical examples are in gitrev.f90 and split_string.f90. For split_string.f90, note that this is the proper way to avoid assumed-length character functions, which are obsolete as of Fortran 95. Specifically:

Bad:

! don't do this!
character(*) function myfun(a)

Good:

pure function myfun(a)
character(:), allocatable :: myfun

Gfortran 5

  • OpenMP 4.0 fully supported
  • Fortran 2003: ieee_ intrinsics supported, allowing convenient standard-compliant use of nan, ieee_is_nan, etc.
  • Fortran 2008: Initial coarray support
  • Fortran 2008: error stop in pure procedures, but only without specifying std= (until Gfortran 8 std=f2018). Error code must be constant.

Gfortran 4.9

  • Fortran 2003: allow deferred-length character variables in derived types

Gfortran 4.8

By this release, substantial Fortran 2008 polymorphism support had been initially added, including

  • select type
  • class(*)
  • type(*)
  • assumed rank dimension(..)

Gfortran 4.6

Gfortran 4.6 is the first version of Gfortran basically usable with commonly used code beyond Fortran 95.

  • Fortran 2003: deferred-length character variable (not in derived types until 4.9)
  • Fortran 2008: impure elemental support

Gfortran 4.5

  • Fortran 2008: iso_fortran_env real64 etc. added

Gfortran 4.4

Gfortran 4.4 added initial support for polymorphism.

  • OpenMP 3

Operating system vs. Gfortran version:

Here are a few common operating systems vs. easily available Gfortran version. CentOS should use devtoolset.

Cygwin

Homebrew

Ubuntu / Debian

  • Ubuntu gfortran repo defaults

  • Get recent Gfortran via PPA

  • Ubuntu 20.04 default: gfortran-9

  • Ubuntu 18.04 default: gfortran-7

  • Ubuntu 16.04 default: gfortran-5

  • Debian Buster, including Raspberry Pi: gfortran-8

CentOS / RHEL

  • release 8 (EOL 2029): gfortran-8

  • release 7 (EOL 2024) devtoolset-8: gfortran-8

  • release 7 EPEL: gfortran-4.9

  • release 7 default: gfortran-4.8

CMake

CMake allows switching parameters based on compiler version. This is very useful for modern Fortran programs.

Fortran 2018

Example CMakeLists.txt for Fortran compiler version dependent options.

if(CMAKE_Fortran_COMPILER_ID STREQUAL GNU)

  # option applying to any language for this compiler
  add_compile_options(-mtune=native)

  # language-specific, note LEADING space
  string(APPEND CMAKE_Fortran_FLAGS " -fimplicit-none")

  if(CMAKE_Fortran_COMPILER_VERSION VERSION_GREATER_EQUAL 8)
    string(APPEND CMAKE_Fortran_FLAGS " -std=f2018")
  endif()

endif()

add_executable(myprog main.f90)

Non-Fortran 2018 workaround

Here we assume that assert.f90 contains error stop in a procedure that’s called from a pure procedure.

include(CheckFortranSourceCompiles)
check_fortran_source_compiles("character :: a; error stop a; end"
  f18errorstop SRC_EXT f90)

add_executable(myprog PRIVATE main.f90)

if(f18errorstop)
  target_sources(myprog PRIVATE assert.f90)
else()
  target_sources(myprog PRIVATE assert_old.f90)
endif()

Notes

Install Nvidia HPC free C, C++, Fortran compilers

The free-to-use Nvidia HPC SDK compiler executables are:

  • C: nvc
  • C++: nvc++
  • Fortran: nvfortran

Existing toolchains such as CMake and Meson can immediately use HPC SDK by setting environment variables:

CC=pgcc
CXX=pgc++
FC=pgfortran

first ensuring the HPC SDK bin/ directory is on PATH.

Nvidia HPC binaries can offer speed improvements over GNU GCC / GFortran, but Intel oneAPI binaries can be significantly faster than Nvidia-compiled binaries for CPU-only workloads. Unless one specifically needs the GPU features of Nvidia HPC SDK consider GNU or Intel oneAPI that have more modern Fortran features.

Nvidia HPC compilers are currently available for Linux, with language standard support:

  • C11
  • C++17
  • Fortran 2003, with some Fortran 2008 including submodule and error stop

Install

  1. Download and install Nvidia HPC SDK:
  2. sudo is not required, but the install must be on symbolic-link-aware drive (not ExFAT)
  3. Add the Nvidia “bin” directory to your “~/.bashrc” PATH environment variable.
  4. Open a new terminal to use Nvidia compilers.

CMake

Currently, CMake can recognize Nvidia compilers as “PGI” since Nvidia HPC SDK takes over for deprecated PGI compilers.

Set compiler-specific options in CMakeLists.txt for various Fortran compilers like:

project(myproj Fortran)

if(CMAKE_Fortran_COMPILER_ID STREQUAL PGI)
  add_compile_options(-Mdclchk)
endif()

Until CMake is updated to recognize Nvidia HPC as a distinct compiler suite, select the Nvidia compilers at CMake configure step:

FC=pgfortran CC=pgcc CXX=pgc++ cmake -B build

sed one-liners to clean blanks

Using sed one-liners, recursively clean from text files:

  • blank lines
  • trailing whitespace

NOTE: ensure the globbing pattern is only for the expected text files or you might goof up PDF files etc. by just using “*”

The script below is used like:

./clean.sh ~/my_site "*.md"

clean.sh contains:

#!/bin/bash

set -e

loc=$1
pat=$2

find $loc -not -path "*/.git*" -type f -name "$pat" -execdir sed --in-place 's/[[:space:]]\+$//' {} \+ -execdir sed --in-place -e :a -e '/^\n*$/{$d;N;};/\n$/ba' {} \+

Note that each “-execdir” command is separate. Add more commands or take out what is unwanted.

Use cases

  • keep files “Git clean” of trailing spaces and extra lines at end of file
  • Matlab editor doesn’t autoclean these lines, so use this script for “*.m” files

Windows SSH server

Since Windows 10 1809, OpenSSH client and server are built into Windows. The setup procedure is easier than using Cygwin. RDP (Remote Desktop) over SSH can be significantly more secure than RDP alone, assuming SSH is well configured.

  1. Enable OpenSSH Server: Go to Windows Settings → Apps → Apps & features → Optional features → Add a feature → OpenSSH Server. This also sets Windows Firewall to allow inbound SSH TCP connections. 2 . Edit c:/ProgramData/ssh/sshd_config on the OpenSSH server PC. At least set PasswordAuthentication no to require SSH public key for better security.

  2. A minimal SSH keypair can be created for the SSH client by:

    ssh-keygen -t ed25519 -f ~/.ssh/my_server
    
  3. Copy the contents of client laptop file ~/.ssh/my_server.pub to the Windows SSH server computer, creating or adding a line to file ~/.ssh/authorized_keys. The location of this file is defined in sshd_config as AuthorizedKeysFile. Use a unique key for each connecting client–do not reuse SSH keypairs between servers or clients.

  4. if the user is a Windows Administrator on the OpenSSH server computer, add the SSH public key to c:/ProgramData/ssh/administrators_authorized_keys

  5. Start the SSH server (for this session only) from PowerShell:

    Start-Service sshd
    

    If this gives an error and/or you wish to always start OpenSSH, type services.msc and in Properties of OpenSSH server → General set “Startup Type: Automatic”

  6. As on Linux, the “authorized_keys” file must have the correct file permissions ACL. Run this PowerShell script from Concurrency.com:

  7. now the SSH client should be able to connect to the SSH server. If this doesn’t work, try using SSH locally on the OpenSSH server computer to troubleshoot.

  8. To use RDP (remote desktop) over SSH do this one-step setup

Tips

  • Edit text files from Windows console over SSH in the Terminal by using WSL:

    wsl
    

    then enter commands like nano foo.txt just like in Linux as it’s the WSL shell.

  • Change the default SSH shell. Assuming you have PowerShell 7 on the SSH server, the commands would be like (from pwsh PowerShell):

    New-ItemProperty -Path "HKLM:\SOFTWARE\OpenSSH" -Name DefaultShell -Value "C:\Program Files\PowerShell\7\pwsh.exe" -PropertyType String -Force
    

print compiler macro definitions

Compilers define macros that can be used to identify a compiler and platform from within C, C++ and Fortran code. This can be useful for many purposes where short bits of platform-specific or compiler-specific code is needed. If a significant amount of code is needed, it may be better to swap in different code files using the build system instead of lengthly #if defined(foo) logic. There are numerous examples for C and C++ so here we will focus on macros of Fortran compilers.

Gfortran

Gfortran compiler macro definitions are obtained in an OS-agnostic way by:

echo "" | gfortran -dM -E - > macros.txt

that creates a file “macros.txt” containing all the compiler macros.

commonly used macros to detect operating system / compiler configuration include:

  • _WIN32 1
  • __linux__ 1
  • __unix__ 1
  • __APPLE__ 1

Intel oneAPI

Intel oneAPI compiler macros include the Gfortran macros noted above and additionally:

  • __INTEL_COMPILER 1

Nvidia HPC Fortran

Nvidia HPC Fortran compiler macros are printed by:

nvfortran -dM

the Nvidia macros include the Gfortran macros above as well as:

  • __PGI 1

Flang

Flang macros include

  • __FLANG 1

Other compilers

Other Fortran compiler macros that identify the compiler and platform can be found in CMake source code.

MPI 3 Fortran 2008 interface

We recommend Fortran MPI programs use the MPI-2 interface via:

use mpi

or even better, the Fortran 2008 MPI-3 interface via:

use mpi_f08

instead of the deprecated and not recommended:

include 'mpif.h'

Intel MPI

IntelMPI supports MPI Fortran 2008 use mpi_f08 including on Windows using free Intel oneAPI compiler.

Interfacing MPI 3 with legacy Fortran code

MPI 3 constants like mpi_comm_world and mpi_real are no longer integer data type, but rather derived types. Some libraries have not yet updated to be polymorphic for this enhanced MPI 3 variable type. For such libraries, access the integer value via the %mpi_val property.

use mpi_f08

integer :: comm = mpi_comm_world%mpi_val
!! %mpi_var emits the legacy integer

Notes

MPI-3

CMake CTest single command build and test

Many CMake users have muscle memory for the three-stanza configure, build, test with CMake:

cmake -B build  # configure CMake and generate build files

cmake --build build  # compile and link binaries

cd build
ctest  # run program self-tests

However, this can be reduced to a single command for many programs:

ctest -S setup.cmake

where a single file setup.cmake is added to the project.

Option selection

If you need to specify command-line build options -Dfoo then either specify them in setup.cmake by _opts variable, or do like:

cmake -B build -Dfoo

ctest -S setup.cmake

Often we have too much data that is still not enough

This example uses the aurora, which is produced around most planetary bodies due to energetic particle kinetics as the particles penetrate the ionosphere. Optical instruments such as cameras give a line integrated measurement for each pixel (angle) of the imagers. This data can be useful for tomographic techniques, when the location and orientation of the camera is well known, and multiple cameras with overlapping field of view exist.

However, this rich data can be greatly supplemented and even superseded by other instruments, especially incoherent scatter radar, where 3-D + time data are available due to volume integrated target returns. Many analyses rely on those thin (~ 0.5 degree FWHM) radar beams to complete an analysis. We rarely know the needed orientation of the radar beams beforehand, and many ISR cannot change the location of their pre-programmed beams. Although as AESA they can steer almost instantaneously within the radar backend processor limits.

This is just a geospace example of like Twitter having too much data, but not enough to gauge individual analyses without additional processing techniques.