Scientific Computing

GoogleTest internally set CMake compiler flags are too aggressive for end users, and can cause build errors. We experienced this with Intel oneAPI, and created a workaround for the GoogleTest-consuming CMake project to override the offending flags. This technique is useful in general with third-party CMake projects, including those obtained by FetchContent.

For GoogleTest, we determined that Intel oneAPI was experiencing nuisance internal errors:

• Windows “-WX” flag
• Linux “-Werror” flag

We overrode those flags with this CMake script:

FetchContent_MakeAvailable(googletest)

if(CMAKE_CXX_COMPILER_ID STREQUAL "IntelLLVM")
  foreach(t IN ITEMS gtest gtest_main gmock gmock_main)
    if(WIN32)
      # necessary since GoogleTest injects /WX blindly and that fails builds with modern IntelLLVM.
      # https://learn.microsoft.com/en-us/cpp/build/reference/compiler-option-warning-level
      target_compile_options(${t} PRIVATE $<$<COMPILE_LANGUAGE:CXX>:/WX->)
    else()
      # necessary to avoid
      # error: unknown warning option '-Wno-implicit-float-size-conversion' [-Werror,-Wunknown-warning-option]
      target_compile_options(${t} PRIVATE $<$<COMPILE_LANGUAGE:CXX>:-Wno-error=unknown-warning-option>)
    endif()
  endforeach()
endif()

The override “/WX-” nullifies the /WX flag that errors for nuisance warnings internal to GoogleTest. The “-Wno-” flag is the same as the underlying GCC compiler on Linux.

CMake build targets are declared by “add_[executable,library,custom_target]” commands. Targets can be dynamically set by arbitrarily complex foreach(), if(), etc. logic. A list of CMake targets in the directory scope is retrieved by the BUILDSYSTEM_TARGETS directory property.

The variable “target_names” contains all the target names previously added in the CMakeLists.txt in the DIRECTORY scope. Retrieving the list of targets in a whole project, or in a FetchContent dependency is possible with this CMake function:

function(print_targets dir)
    get_property(subdirs DIRECTORY "${dir}" PROPERTY SUBDIRECTORIES)
    foreach(sub IN LISTS subdirs)
        print_targets("${sub}")
    endforeach()

    get_directory_property(targets DIRECTORY "${dir}" BUILDSYSTEM_TARGETS)
    if(targets)
        message("Targets in ${dir}:")
        foreach(t IN LISTS targets)
            message("  • ${t}")
        endforeach()
    endif()
endfunction()

Use this function like:

print_targets("${CMAKE_CURRENT_SOURCE_DIR}")

Or supposing FetchContent, here using “googletest”:

FetchContent_MakeAvailable(googletest)

FetchContent_GetProperties(googletest)
print_targets(${googletest_SOURCE_DIR})

results in:

Targets in _deps/googletest-src/googletest:
  • gtest
  • gtest_main
Targets in _deps/googletest-src/googlemock:
  • gmock
  • gmock_main

Note: get_property(test_names GLOBAL PROPERTY BUILDSYSTEM_TARGETS) will return an empty list–DIRECTORY scope must be used.

Windows Subsystem for Linux (WSL) uses VHDX files to store each distribution’s filesystem. Over time these disk images grow in size as files are added and deleted. However, the space used by deleted files is not automatically reclaimed, leading to larger disk images than necessary.

There is a PowerShell script to automatically compact WSL VHDX files that works with any of the Windows release levels (including Home).

The jpegtran command line tool allows lossless transformations on JPEG images, including rotation, cropping, and flipping. This is particularly useful when you want to modify JPEG images without re-encoding them, which can lead to quality loss.

Example: lossless clockwise 90 degree rotation of a JPEG image “input.jpg” and save the result to file “output.jpg”:

jpegtran -rotate 90 input.jpg > output.jpg

WiFi captive portals and public networks often block outbound network port traffic. Sometimes even VPNs are blocked. Often only ports 80 (HTTP) and 443 (HTTPS) are allowed.

For Git, one can use Git with HTTPS and Oauth tokens instead of Git over SSH. However, note that Git over SSH has certain benefits for ease of use and security.

To quickly determine if outbound network ports are blocked, portquiz.net is a useful free service. Using Python automates this process for multiple ports concurrently using concurrent.futures.ThreadPoolExecutor threads or asyncio. We provide an example of each method in short scripts. The examples shows that Asyncio using Semaphore with AIOHTTP can be faster than ThreadPoolExecutor with urllib.request.

Solutions to blocked ports for SSH include using SSH ProxyJump with an intermediate TRUSTED server on an allowed port. Some remote SSH systems actually require this, where they the desired server has only LAN access, and a gateway SSH server with no privileges is used as the SSH ProxyJump by network design.

The ultimate workaround would be a mobile hotspot (different network).

LaTeX code highlighting is possible using the minted LaTeX package, which uses Pygments as a backend. Minted is available in Overleaf as well.

If needed, install minted via TeX Live Manager (tlmgr)

tlmgr install minted

Install Python Pygments:

python -m pip install pygments
# or
brew install pygments

If using a GUI, the --shell-escape option may need to be added. In TeXmaker / TeXstudio, under “Options → Configure → Commands” add --shell-escape right after the compiler executable like xelatex --shell-escape or pdflatex --shell-escape or latexmk --shell-escape.

If “pygmentize” isn’t found, under TeXstudio Preferences → Build, check Show Advanced Options in the lower left checkbox and set Additional Search Paths → Commmands ($PATH) to the directory containing “pygmentize”, e.g. /opt/homebrew/bin on macOS with Homebrew or whatever directory comes up for which pygmentize in a Terminal.

In general operation systems set a limit to the number of open files per process. A “file” might be a regular file, a socket, a pipe, etc. When a Python program exceeds this limit, it raises exception:

OSError: [Errno 24] Too many open files

Such issues might tend to arise with highly concurrent applications as enabled by asyncio or higher-level libraries. Resolving such asyncio concurrency issues generally involves asyncio primitives like asyncio.Semaphore. Implementing asyncio libraries correctly is so non-trivial that using higher-level libraries that implement such concurrency control correctly can be a better idea. It’s important to go beyond toy examples and consider real-world concurrent Python usage patterns.

Certain projects with legacy build systems like Make or Autotools or in general may specify to build with flags like “-fPIC” for position independent code (PIC) on Linux systems. Consider not forcing these flags in CMake projects if there isn’t a specific known need, to let users and consuming projects decide whether they need PIC or not. When PIC is needed, do like:

if(...)
  include(CheckPIESupported)

  check_pie_supported()

  set(CMAKE_POSITION_INDEPENDENT_CODE true)
endif()

The check_pie_supported module checks whether the compiler supports PIE and should be run before setting target property POSITION_INDEPENDENT_CODE.

When PIC is enabled with certain platforms like Intel oneAPI on Linux, linker errors may result like

relocation R_X86_64_32 against `.rodata.str1.1’ can not be used when making a PIE object

where the solution is to disable PIC – or simply do not enable it by default in the CMake project as we suggested above.

Since CMake 3.13 was released in 2018, a more elegant two-step CMake build process is available with the simple syntax

cmake -B ./build

cmake --build ./build

The CMake configure command with the “-B” option generates the build system files in the specified build directory. It’s best to build out-of-source in general, so we specify a separate “./build” directory here.

The cmake –build build command executes the build tool configured for the project, e.g. Make, Ninja, MSBuild, etc.

The “./” can be omitted, we only include it here for clarity. We keep seeing even the largest OEMs using the old-fashioned three-step process. Check your projects’ example and docs to update them to the less-confusing and compact two-step CMake build process.

In CMake projects (or other build systems) it’s more robust to detect compiler features rather than make brittle, difficult to maintain nested if / elseif logic trees based on compiler vendor and version. There are edge cases where if() statements to work around known compiler bugs are necessary, but where we can, we use compiler feature checks instead. Each compiler vendor such as AppleClang, GNU GCC, LLVM Clang, typically publishes tables of compiler features vs. language standard support in their documentation. These tables might not cover every aspect of feature support, or the project invocation of that support may trigger bugs in specific compiler versions. Then, either hard-coded build system logic or preferably configure-time feature detection is needed.

An example using check_source_compiles in a CMake project requires C11 variable length arrays. The CMakeLists.txt would look like:

include(CheckSourceCompiles)

set(CMAKE_C_STANDARD 11)
set(CMAKE_C_STANDARD_REQUIRED ON)
# so that check_source_compiles sets the correct language standard

set(CMAKE_TRY_COMPILE_TARGET_TYPE STATIC_LIBRARY)
# save link time, only compile is needed

check_source_compiles(C "int main(void){
for (int i = 1; i < 5; i++){
  int a[i];
}
return 0;
}"
c11vla)

if(NOT c11vla)
 # fatal error or disable feature using C11 VLA
endif()

add_executable(c11demo demo.c)

A similar meson.build configure-time compiler feature check example:

project('c11vla_demo', 'c',
  default_options : ['c_std=c11'])

c11_vla = meson.get_compiler('c').compiles(required: true, name: 'c11_vla',
  'int main(void){ for (int i = 1; i < 5; i++){ int a[i]; } return 0; }'
)