Scientific Computing

Gfortran 10 added the default behavior to make type mismatches an error instead of a warning. Legacy Fortran programs too often did not explicitly specify the procedure interfaces to allow implicit polymorphism. The Fortran 2008 standard cleaned up this situation in part with type(*).

Workaround: Gfortran flag -fallow-argument-mismatch can be used to degrade the errors to warnings. It is however strongly recommended to fix the problem in the legacy code, if it’s part of your code ownership.

Possible workaround

For external libraries like MPI-2, the interfaces are intended to be polymorphic but use Fortran 90-style interfaces. The user code can declare an explicit interface.

However, this is not recommended – we have seen intermittent runtime errors with MPI-2 using the technique below, that were entirely fixed by using the “mpi_f08” MPI-3+ interface.

use mpi, only : MPI_STATUS_SIZE

implicit none

interface
!! This avoids GCC >= 10 type mismatch warnings for MPI-2
subroutine mpi_send(BUF, COUNT, DATATYPE, DEST, TAG, COMM, IERROR)
type(*), dimension(..), intent(in) :: BUF
integer, intent(in) ::  COUNT, DATATYPE, DEST, TAG, COMM
integer, intent(out) :: IERROR
end subroutine

subroutine mpi_recv(BUF, COUNT, DATATYPE, SOURCE, TAG, COMM, STATUS, IERROR)
import MPI_STATUS_SIZE
type(*), dimension(..), intent(in) :: BUF
integer, intent(in) ::  COUNT, DATATYPE, SOURCE, TAG, COMM
integer, intent(out) :: STATUS(MPI_STATUS_SIZE), IERROR
end subroutine

end interface

Many code languages don’t have garbage collection and also suffer from issues like uninitialized variables etc. CMake can use several common tools to check memory usage of a program. These tools are limited by the code coverage of the tests run.

The basic procedure is to build the project in “Debug” mode, run CTest memcheck task, then browse the results. Then fix the problems and rerun the tests to verify.

The popular “valgrind” memory tester is available on Linux, including Windows Subsystem for Linux (WSL). CMake will usually automatically detect valgrind if the valgrind executable is on the environment variable PATH. That is, if typing “valgrind” at the Terminal runs, then CMake should find and use it.

To use memory checking tools from CMake, the CMakeLists.txt must include the line:

include(CTest)

Here are example CMake commands to check memory of a project.

cmake -Bbuild -DCMAKE_BUILD_TYPE=Debug
cmake --build build

ctest -T memcheck --test-dir build

The tests may run more slowly than usual due to the testing overhead. The CTest terminal output will look like:

MemCheck log files can be found here: (<#> corresponds to test number)
build/Testing/Temporary/MemoryChecker.<#>.log
Memory checking results:
Memory Leak - 8
Potential Memory Leak - 8
Uninitialized Memory Conditional - 3

Each MemCheck CTest number has a log file. The traceback stack to the suspected area (line) of code detected to lead to memory problems is given. Edit the code to fix the problem, then rebuild and rerun memcheck as above.

Configure CTest memcheck

To change from default CTest memcheck settings, create a script memcheck.cmake in the top level of the project (where the top CMakeLists.txt is). Run this script like:

ctest -S memcheck.cmake -V

This program also has an example valgrind suppressions file. The suppressions file is derived from valgrind option

valgrind --gen-suppressions=all <executable>

It’s often useful to set per-test environment variables in CMake’s CTest testing frontend. The environment variables appear and disappear with the start and end of each test, in isolation from any other tests that may be running in parallel. This is accomplished via CMake test properties ENVIRONMENT and ENVIRONMENT_MODIFICATION.

Example: set environment variable FOO=1 for a test “bar” like:

set_property(TEST bar PROPERTY ENVIRONMENT "FOO=1")

multiple variables are set with a CMake list (semicolon delimited) like:

set_property(TEST bar PROPERTY ENVIRONMENT "FOO=1;BAZ=0")

Here comes an issue. In general, Windows needs DLLs to be on the current working directory or in environment variable PATH. We handle this by a script that appends to PATH for CTest on Windows:

In Python likewise set/unset environment variables within tests using PyTest monkeypatch fixture.

Package managers can detect which packages were manually installed (at user or script explicit request) and which were implicitly installed as a prerequisite. When uninstalling the manually installed package, the prerequisite packages are often not auto-uninstalled. To recover a significant amount of disk space (gigabytes perhaps) from unused packages, an autoremove command is useful.

Pacman is used in numerous Linux distros and in MSYS2. Show the auto-installed prerequisites:

pacman -Qdtq

This can be piped into the Pacman remove command upon verifying the packages above are indeed OK to remove:

pacman -Qdtq | pacman -Rs -

The package cache can contain gigabytes no longer needed. The pacman cache is cleared by:

pacman -Sc

Related: Clean APT and DNF cache

mdadm is a popular Linux tool to manage RAID arrays. RHEL provides a usage guide.

After creating a RAID array, or to find an existing array that may or may not be mounted, use commands like:

mdadm --detail /dev/md/Volume0_0

Note the individual RAID disk /dev/sd*, which can be further examined like:

mdadm --examine /dev/sda /dev/sdb

Also examine all available devices with:

fdisk --list

To mount the RAID, use commands like:

mkdir /mnt/raid
# arbitrary location you want to mount RAID device at

mount /dev/md/Volume0_0p1 /mnt/raid

There might be extra devices under /dev/md that can’t be mounted, but one of them should be the desired RAID.

CMAKE_LINK_LIBRARY_USING_FEATURE doesn’t have a feature to delay loading import for MSVC-like compilers flag like /delayload. Nonetheless, /delayload can be accomplished in a compact way as in the following example:

add_library(example SHARED lib.c)
add_executable(main main.c)

target_link_libraries(main PRIVATE example)

if(MSVC)
  set_property(TARGET example PROPERTY WINDOWS_EXPORT_ALL_SYMBOLS true)
  target_link_libraries(main PRIVATE delayimp)
  target_link_options(main PRIVATE "/DELAYLOAD:$<TARGET_FILE_BASE_NAME:example>.dll")
endif()

FFmpeg can losslessly copy the video stream while removing the audio stream. Use a command like:

ffmpeg -i input.mp4 -an -c:v copy output.mp4

-an

omits the audio stream from the output file

-c:v copy

losslessly copy the video stream without re-encoding it

CMake configure time is often a convenient time to test if a directory or file is writable. It can be useful to immediately stop with message(FATAL_ERROR) if a path is not writable. For example, if CMAKE_INSTALL_PREFIX is not in a writable location – we want to fail right then and inform users what to do to correct the problem.

file(TOUCH) and file(MAKE_DIRECTORY) do not halt the configure step. Rather, if a path isn’t writable CMake only errors at the end of configure.

Example solution:

This snippet generates a fatal error with text telling the user what to try:

set(test_path /path/to/test/.ignore)
# could be done with random string filename in the desired path

execute_process(COMMAND ${CMAKE_COMMAND} -E touch ${test_path}
RESULT_VARIABLE ret
)
if(NOT ret EQUAL "0")
  message(FATAL_ERROR "No write access to ${test_path}
  <text for user to resolve issue>")
endif()

SSL certificate checking can add security to web operations. Some systems may need environment variable SSL_CERT_FILE for Matlab’s vendored curl. As a last resort, certificate checking can be turned off, but this raises file integrity and security issues.

Instead of disabling certificate checking set environment variable SSL_CERT_FILE to the actual certificate location.

Matlab or GNU Octave use the factory function weboptions() to control HTTP behavior for functions like websave() and webread(), including Timeout and SSL certificate.

This example sets reply timeout to 15 seconds and specifies custom SSL certificate location when environment variable SSL_CERT_FILE is set.

cert = getenv("SSL_CERT_FILE")

if isfile(cert)
  web_opts = weboptions(CertificateFilename=cert, Timeout=15);
else
  web_opts = weboptions(Timeout=15);
end

data = webread(url, opts);

Alternative: curl with Matlab

Related: Git SSL certificate location

Connecting to HTTPS servers with curl or programs using curl such as Matlab requires curl knowing the location of system certificates. If curl doesn’t know the certificates location, accessing HTTPS URLs may fail with:

curl: (77) error setting certificate verify locations: curl can't find your certificates.

Fix this problem by setting environment variable “SSL_CERT_FILE” to the location of the system certificates. Example file locations include:

/etc/ssl/certs/ca-bundle.crt
/etc/ssl/certs/ca-certificates.crt
/etc/pki/tls/certs/ca-bundle.crt
/etc/ssl/ca-bundle.pem

Reference

Related: Git SSL certificate location