OpenMPI 3 Fortran 2008 interface update

In legacy Fortran MPI programs, near the top of the procedure we would have:

use mpi

or

include 'mpi.h'

It is typically recommended to use the OpenMPI Fortran 2008 interface:

use mpi_f08

Intel MPI

Intel MPI supports MPI Fortran 2008 use mpi_f08 since Intel 16.0 EXCEPT for Windows Intel MPI, where even Intel 19.0 did not have mpi_f08.

Interfacing MPI 3 with legacy Fortran code

You will notice for example that constants like mpi_comm_world and mpi_real are no longer integer data type, but rather custom types.

Many legacy and even current libraries (such as MUMPS 5) have not yet updated to be polymorphic for this enhanced variable type.

Here is a minimal working example for MUMPS using the backwards compatible OpenMPI 3 interface, grabbing the integer via the %mpi_val property.

program mumps_mpi3

use mpi_f08
include 'dmumps_struc.h'  ! per MUMPS 5 manual

type(dmumps_struc) mumps_par

mumps_par%comm = mpi_comm_world%mpi_val   ! %mpi_var emits the legacy integer


end program

Notes

OpenMPI 3

Five free C C++ Fortran compiler families

Dozens of “free” compilers exist for C, C++ and Fortran. However, only five compiler families available at no charge support the modern features of these popular compiled languages. This list covers four compiler families available at no charge that support most of:

  • C17
  • C++17
  • Fortran 2008

GCC

GCC has broad support of modern standards on a very wide range of computing platforms. GCC’s downside in some cases can be slower runtime performance than compilers having less broad language and platform support.

LLVM

LLVM Clang and Flang have significant industry support, including from Nvidia, and are known for high performance, but somewhat less language feature support and less broad platform support than GCC.

PGI

PGI Community Edition is free to use, and has performance comparable to Intel compilers in many cases. PGI ≥ 19 should be used for modern Fortran. PGI supports CUDA Fortran.

Intel

Intel compilers are a paid product in general. For non-commercial purposes, including:

  • educators
  • students
  • open-source projects

the Intel compilers are available at no charge. The Intel performance libraries like MKL, IPP, TBB and more are available at no cost with more liberal use terms.

IBM XL

IBM XL compilers are currently for POWER CPUs only e.g. ppc64le. IBM XL compilers do not work with a typical x86-based computer. If you have a $2000 Raptor IBM POWER9 desktop, then IBM XL may be for you.

The IBM XL compilers are high-performance compilers that have a free community edition. IBM XL Fortran has wide support for Fortran 2008.

Setup Docker Fortran image for Travis-CI

The procedure described here is for a new user to Docker. There are certainly more efficient ways to do these tasks that are perhaps less easy for less experienced Docker users. Please let us know if you have a suggested improvement suitable for novice Docker users.

Install Docker image

  1. add docker group and yourself to this group, to avoid needing sudo for every docker command:

    addgroup docker

adduser $(whoami) docker

    then reboot (not just logout, an actual system restart)

  2. install Docker on your laptop (the example here is for a Linux laptop)

    snap install docker

    after this step, sudo should no longer be required.

  3. try the Hello World images, which should auto-download and print a welcome message

    docker run hello-world

  4. Search for a desired image. Consider those images listed as “official”. You might need to widen your terminal to 155 columns or so to render properly. Let’s use ubuntu as it’s quite popular for Docker and in general.

    docker search ubuntu

  5. get the desired image

    docker pull ubuntu

  6. verify the images on your computer

    docker images

    ubuntu takes just under 100 MB to start.

  7. start the Docker container on your laptop

    docker run -it ubuntu

    where -it means interactive session verify the version running, it will have a # superuser prompt:

    cat /etc/lsb-release

manage containers

These commands are issued NOT from within the Docker container itself (open a new terminal)

  • where are containers stored: docker info | grep "Root Dir"
  • How much space is Docker using altogether: du -sh /var/snap/docker
    • use previous command to find where your Docker stuff is
  • list images: docker images -a
  • list containers (running and stopped): docker ps -a
  • stop a Docker container: docker stop container_name
  • start a Docker container: docker start container_name

  • login to a running Docker container: docker exec -it container_name bash

  • get container environment variables: docker exec container_name env

  • cleanup unused containers docker system prune

  • delete image: docker image rm ubuntu:19.04

Configure Docker image

At least through April 2019, Travis-CI only had up to Ubuntu 16.04 dist: xenial, which does not have OpenCoarrays or other contemporary software that’s in Ubuntu 18.04 Bionic. Travis-CI (and countless other online, server and computer resources) can use Docker images. For research reproducability, a Docker image gives a known software state that should be usable for many years.

Interactive setup method tends to make the final image larger than using RUN commands. So let’s start off by using a Dockerfile as that helps maintain smaller images with repeatable builds. The Docker container can be turned into a new image that is usable from Travis-CI or any other suitable Docker host.

We do not claim this Dockerfile to be optimal. Please let us know of suggested improvements.

# based on https://github.com/naftulikay/docker-bionic-vm/blob/master/Dockerfile

FROM ubuntu:18.04
ENV container=docker TERM=xterm LC_ALL=en_US LANGUAGE=en_US LANG=en_US.UTF-8

# stop some annoying interactive prompts, also need "-yq" on apt-get
ENV DEBIAN_FRONTEND=noninteractive

# locale
RUN apt-get update -q > /dev/null && \
  apt-get install --no-install-recommends -yq apt-utils locales language-pack-en dialog > /dev/null && \
  locale-gen $LANGUAGE $LANG

# add sudo commmand
RUN apt-get -yq install sudo > /dev/null

# create and switch to a non-priviliged (but sudo-enabled) user, arbitrary name
RUN echo "nonprivuser ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers
RUN useradd --no-log-init --home-dir /home/nonprivuser --create-home --shell /bin/bash nonprivuser && adduser nonprivuser sudo
USER nonprivuser
WORKDIR /home/nonprivuser

# Git/Curl -- don't disable recommends here or you won't have Certification Authority certificates and will fail
RUN sudo apt-get install -yq git curl > /dev/null

# packages specific to your needs
RUN sudo apt-get install --no-install-recommends -yq make gfortran libcoarrays-dev libopenmpi-dev open-coarrays-bin > /dev/null && \
  sudo apt-get clean -q

# latest cmake
RUN git clone --depth 1 https://github.com/scivision/cmake-utils && \
  mkdir -v /home/nonprivuser/.local && \
  cd cmake-utils && PREFIX=/home/nonprivuser/.local ./cmake_setup.sh > /dev/null  && \
  mv -v /home/nonprivuser/.local/cmake* /home/nonprivuser/.local/cmake

ENV PATH=$PATH:/home/nonprivuser/.local/cmake/bin

# other optional installs

# RUN sudo apt-get install --no-install-recommends -yq octave
  1. create the file above named Dockerfile.

  2. build the Docker image:

    docker build -t opencoarrays_fortran .

    assuming you’re in the same directory as Dockerfile

  3. check the image exists, for me it was about 350 MB:

    docker images

  4. Before committing for upload, you must invoke the container.

    docker run opencoarrays_fortran

    This will almost immediately start and stop, as you didn’t give it a command to run or persist.

  5. get the hexadecimal container ID by:

    docker ps -a

    the container will have an auto-assigned name like foo_bar. Note the hexadecimal “container ID”.

upload Docker image

Once you have configured a Docker container on your laptop suitable for your purposes, you may wish to share this container and use it on other hosts such as Travis-CI. This can be done for free with DockerHub.

Once you are ready to upload the image, note the “container ID”, which is the hexadecimal number at the terminal prompt of the Docker container, under docker ps. The container hex ID must appear under docker ps, just being in docker images is not enough.

docker commit -m "basic fortran OpenCoarrays setup" hex_id dockerhub_username/opencoarrays_fortran

The changes to the image the container made are gathered into a new image. It may take a minute or two if your image is large. Ideally with a small image it will take only a couple seconds. The new image has not yet left your computer, it will show up under

docker images

Once uploaded, your Docker image is visible to the world by default. {: .alert-box.warning}

  1. Login to DockerHub

    docker login -u dockerhub_username

  2. Push (upload) this image. Note this image is world-visible by default!

    docker push dockerhub_username/opencoarrays_fortran

If your image is very large > 1 GB, be aware it will take a while to upload, and for the CI system to download. This is a telltale that it’s important to keep Docker images small.

Setup Travis-CI to use this image

Create .travis.yml:

language: minimal

services: docker

git:
  depth: 3
  quiet: true

env: Dimg=scivision/opencoarrays_fortran; Dname=fortran-ubuntu

before_install:
# Travis UID does not necessarily match 1000:1000 of Docker FIXME will Docker always be 1000:1000
- sudo chown -R 1000:1000 $TRAVIS_BUILD_DIR

- docker run -d --name $Dname -v $TRAVIS_BUILD_DIR:/home/nonprivuser/travis -w /home/nonprivuser/travis $Dimg tail -f /dev/null

# prepend $Dcmd to run commands inside the Docker container
- export Dcmd="docker exec -t $Dname bash -c"

install:
# for this example, I already installed everything I need in the image previously created by the Dockerfile

script:
# all commands must be quoted
- $Dcmd "cd build && cmake .. && cmake --build . && ctest -V"

Notes

  • each docker exec command is a new shell instance. So changing directories in one docker exec has no effect on subsequent commands for example. Note the CMake one-liner in .travis.yml that configures, builds and tests in one command.

References

Find files from the command line

One can very rapidly find files with arbitrary criteria on systems with GNU Findutils. This includes Linux, MacOS and Windows Subsystem for Linux.

Examples

Just about any criteria one could think of can be used to rapidly find files. If working on a remote filesystem mounted over SSHFS, we suggest SSHing into that system and running the find command that–it will be orders of magnitude faster.

Most examples use home directory ~ as a starting point just for convenience. Appending 2>/dev/null to the end of a command removes nuisance messages about file permissions. If piping the find command, put 2>/dev/null before the pipe.

Find files with “report” in the filename, case-insensitive

find ~ -iname "*report*"

Suppose ~/data is a symbolic link to another directory, either on your computer, a USB hard drive or a server. By default, find will not search this resource unless you “resolve” the symbolic link to a directory by putting a final / like ~/data/:

find ~/data/ -iname "*report*"

See the findutils manual symbolic link section for more details, in particular the -H and -L options.

Intel MPI on Windows

Compiling with Intel compilers on Windows is a distinctive task from MacOS or Linux. The Intel C, C++ and Fortran compilers masquerade as Visual Studio compilers, especially in their command line options. For the Intel Fortran compiler on Windows, the build options one normally uses for Intel Fortran on MacOS and Linux do not work, and even error out the compiler. For C and C++, one must use the icl compiler (there is no icc or icpc on Windows) as if it were MSVC. That can be perplexing at first for those not coming from the MSVC world. The scientific computing world and most software engineering tasks not originating in the Microsoft domain use GCC-like compilers, which have completely different sets of compiler options.

Loading Intel compiler environment

It’s important that you load the compilervars.bat script to enable the Intel compilers for each session and NOT the psxevars.bat. For convenience, make a batch script like %HOMEPATH%\intel.bat containing:

C:\"Program Files (x86)"\IntelSWTools\parallel_studio_xe_2019\compilers_and_libraries_2019\windows\bin\compilervars.bat intel64`

set FC=ifort
set CC=icl
set CXX=icl

MPI

Another distinctive feature of using MPI on Windows is that a username and password are required, even on a simple standalone laptop. Good security practices may include creating a separate unprivileged user account that is only used for MPI jobs. If this is suitable for your security environment, run MPI jobs in that other user account by from Command Prompt:

runas /user:mympiusername cmd

This opens a new Command Prompt window. The environment variables are not passed to this new windows, so you may need to run Intel compilervars.bat again.

You can register the user credential into the Windows registry, if appropriate for your environment. If doing so, we would again urge to consider using a separate user account that’s only used for MPI runs.

If you don’t sign in, errors will be like:

Unable to manage jobs using credentials with a blank password. Please enter another account.

Intel MPI on Windows is only for Intel compiler

Unlike for Linux Intel MPI that may be used with:

  • GNU gcc and gfortran out of the box
  • PGI via $MKLROOT/../mpi/binding/intel-mpi-binding-kit.tar.gz

Windows Intel MPI is only for the Intel Fortran compiler and none other at this time.

MPI 2008

Intel 19 MPI does not support use mpi_f08 the Fortran 2008 bindings. To keep compatibility with Intel MPI on Windows, at this time one would consider the legacy Fortran MPI interface:

use mpi

call mpi_init(ierr)

call mpi_finalize(ierr)

end program

The Fortran 77 include mpif.h should not be used in current Fortran programs as it breaks the fundamental Fortran 90 paradigm of modularity.

Errors from old Python setuptools

Setuptools 40.6 added section [options.data_files] to setup.cfg. Having setuptools < 40.6 results in this error upon trying to install a Python package:

distutils.errors.DistutilsOptionError: Unsupported distribution option section: [options.data_files]

Fix: upgrade setuptools via:

pip install --upgrade setuptools

Example

A setup.cfg might contain [options.data_files] section like:

[options.data_files]
data = myprogram/data/stuff.txt

Python calling Python via subprocess

On Windows, calling Python sys.executable or console scripts run as .exe may fail with

Fatal Python error: _Py_HashRandomization_Init: failed to get random numbers to initialize Python

The issue arises when environment variables are passed in via the env= argument to Python subprocess. In general, one should add or overwrite variables to the OS environment in subprocess calls as follows in this example for using Clang and Flang compilers. os.environ returns a mapping (general form of dict) of environment variables.

import os
import subprocess

# get a Mapping of all the current environment variables
env = os.environ

# set these to use Clang and Flang compilers
myvar = {'CC': clang, 'CXX': clang++, 'FC': flang}

# %% This is the key section of code--add / overwrite environment variables for this subprocess only.
env.update(myvar)

subprocess.check_call(['meson', 'setup'], env=env)

Summary

Whenever using Python subprocess environment variables, you should generally pass in all the existing environment variables, adding or overwriting the specific variables needed. Otherwise, fundamental environment variables will be missing and the subprocess call generally won’t work. By default, when not specified, env=None, which tells subprocess to copy all the environment variables of the shell Python was called from.

Remove Windows setx environment variables PowerShell

In Windows, the setx command allows storing environment variables in the Windows registry on a per-user or system basis. A problem arises when it is desired to remove such variables entirely. Unlike older versions of Windows, setting a blank value does not work in Windows 10.

Example

Assume one previously set the environment variable CC to the Intel C compiler like:

setx CC icc

Now the problem is, you’d like the system to fall back to using whatever default C compiler is on the Windows system. The existing variable is visible in PowerShell via:

Get-ChildItem Env:CC

But trying to remove the variable with PowerShell command

Remove-Item Env:FC

is only effective for this PowerShell session. The old value of CC comes back upon opening a new PowerShell.

Permanently remove setx variable

To permanently remove the variable, we must remove it from where it’s stored in the Windows registry. For this example of wanting to delete enviornment variable CC, do from PowerShell:

reg delete "HKCU\Environment" /v CC

Optionally, confirm deletion with PowerShell:

Get-ChildItem Env:CC