Scientific Computing

Create RAM drives with open source tools

RAM drives are a filesystem mount point mapped to RAM instead of a hardware storage device. RAM drives are very fast and have high IOPS compared to durable storage devices, which is useful for applications like interprocess communication (when network socket pipes are not usable), temporary files, and caches. Unless synced or copied to durable storage, the contents of RAM drives are ephemeral and may be lost on reboot, logout, or unmounting. We give simple examples of creating / using RAM drives on Windows, Linux, and macOS with open source tools.

Linux RAM drive

On Linux systems, RAM drives are typically built into the OS. Mount points /dev/shm/ and /run/shm/ map shmem shared memory to a RAM drive. They are also present in Windows Subsystem for Linux (WSL).

df -kh /dev/shm
free -h

# Write 1 GB to shmem
dd if=/dev/zero of=/dev/shm/blah bs=10M count=100

df -kh /dev/shm
free -h

See which one has 1GB space more used. This tells if hard drive or RAM was used.

For programs using shared memory for heavy writing operations:

  1. order(s) of magnitude slower /dev/shm/ operations when HDD is used versus RAM
  2. wearing of solid state drive if /dev/shm is pointed there

Example data bandwidths:

  • SSD: 100s of MB/sec
  • HDD: 10s of MB/sec
  • RAM: 1000s of MB/sec

macOS RAM drive

macOS factory program diskutil can create a RAM drive via factory program hdiutil like:

diskutil partitionDisk $(hdiutil attach -nomount ram://2097152) 1 GPTFormat APFS "RAMDisk" '100%'

The command line options are:

partitionDisk
partitions a disk
$(hdiutil attach -nomount ram://2097152)
creates a RAM drive
-nomount
prevents RAM drive from being automatically mounted in Finder
ram://2097152
creates a 1 GB RAM drive (512 bytes/sector * 2097152 sectors = 1 GB)
1
creates 1 partition on the RAM drive
GPTFormat
formats the partition with GPT
APFS
formats the partition with APFS
"RAMDisk"
names the partition “RAMDisk”
'100%'
uses 100% of the RAM drive for the partition

There is a shell utility to simplify the process of creating and managing RAM drives on macOS with command line options. A Swift GUI open source app for those who prefer a graphical interface is also available.

References:

Windows RAM drive

On Windows, RAM drives can be created with the open source tool RamDrive. This is a user-mode RAM drive that can be created and used without administrator privileges, unlike kernel-mode RAM drives such as ERAM.

Update Conda from command line

From time to time generally it’s good to update Conda from Command Prompt / Terminal. Conda itself will remind the user to do this.

conda update -n base -c defaults conda

If it’s been a long time since updating, sometimes instead of updating conda, conda just repeats than an update is available and finally # All requested packages already installed. To workaround this issue, do a complete conda update by:

conda update -n base conda --all

After this, purge old package install info by:

conda clean --all

Install Rust lang on Windows

Even if not directly coding in Rust, one may need to build programs in Rust. For example, to control Apple Studio Display brightness from Windows, one would build the asdbctl Rust program. Many revivals (and newly developed) open-source games are in Rust, such as the classic forest-fire game firefighter.

Rust programs are generally OS-agnostic, unless the Rust code itself uses OS-specific features. Install the Rust toolchain on Windows using WinGet:

winget install --id=Rustlang.Rustup -e

Building and running a Rust program is generally accomplished from the directory containing the Cargo.toml manifest file:

cargo run --release

Intel oneAPI compiler flag syntax

The Intel oneAPI compiler flags are generally somewhat distinct on Windows and Linux. However, there is a platform-agnostic syntax that can be used to specify the flags in a way that works on Windows and Linux without if-else logic needed in the build system (e.g. CMake). The oneAPI OS-agnostic compiler flag syntax is as follows:

-option:value

Where option is the name of the compiler flag and value is the value to be passed to the flag. Observe this is a hybrid of Linux options that use - as a prefix, and Windows options that use : as a separator between the option and its value. This syntax allows developers to write build scripts and command lines that are portable across both Windows and Linux without needing to change the syntax of the compiler flags. This method is used internal to CMake for example, as a demonstration of its robustness.

Fortran Package Manager install

The Fortran Package Manager (fpm) is a tool for building Fortran projects and their dependencies. FPM stands alone, written in Rust, as an alternative to CMake, Meson, or Make for building hierarchies of Fortran projects. The fpm build command is used to build a Fortran project and its dependencies. If a main executable program exists, fpm run runs the executable after building. The fpm install command is used to install a Fortran project and its dependencies. fpm test is used to run tests after building.

FPM itself is easily installed by:

  • Windows winget install FortranLang.fpm
  • macOS: brew install fpm

In general and including for Linux, downloadable FPM binaries are available from the FPM GitHub releases page.

CMake LINKER_LANGUAGE with Fortran and C++

CMake target property LINKER_LANGUAGE is used by CMake to determine the language of the linker when linking a target. When this target property is not set, CMake uses a heuristic to determine the linker language based on the source files and the compiler being used. LINKER_LANGUAGE is important for mixed-language projects, such as those that use both Fortran and C++. However, CMake’s heuristic for determining the linker language can sometimes fail, especially with certain compilers like Intel oneAPI or NVHPC.

This is the general CMake script we use on our Fortran and C++ mixed projects:

set(linker_lang)
if(CMAKE_CXX_COMPILER_ID STREQUAL "AppleClang" AND CMAKE_GENERATOR STREQUAL "Unix Makefiles")
  # otherwise failed to link since -lc++ is missing
  set(linker_lang CXX)
elseif(CMAKE_CXX_COMPILER_ID MATCHES "IntelLLVM|NVHPC" AND NOT CMAKE_GENERATOR MATCHES "Visual Studio")
  # IntelLLVM|NVHPC need Fortran
  set(linker_lang Fortran)
endif()
if(linker_lang)
  message(STATUS "Setting linker language to ${linker_lang} for certain targets")
endif()

Then for individual mixed Fortran and C++ targets, we set the LINKER_LANGUAGE property explicitly:

set_target_properties(mixed_target PROPERTIES LINKER_LANGUAGE ${linker_lang})

Zoom spatial audio - reduce meeting fatigue

Zoom web conference software can enable spatial audio separation for meeting audio. While in the meeting, click the Zoom menubar at the top of meeting windows and select “Gallery” or “Dynamic Gallery”. Enabling spatial audio separation can significantly reduce fatigue for some users, as it allows for spatial separation and localization between speakers, making the audio feel more natural. Spatial audio separation requires at least two speakers, with enough physical separation to make the spatial audio distinguishable. Each speaker’s audio will seem to come from the same left-center-right direction as the relative location of the speaker’s video on the screen. A compact soundbar with closely spaced speakers may lead to unremarkable results. Spatial audio separation is distinct from stereo audio intended for music and media sharing, which is a different feature that allows for stereo audio when sharing music or media content in Zoom meetings.

Numpy Type Annotation

Numpy type annotation numpy.typing.ArrayLike is often used on the input variables of a function. This doesn’t cover all cases a function might accept. For example, numerous Numpy functions can accept Python datetime that becomes an opaque object when passed to a function. It can be better to pass datetime into Numpy functions as numpy.datetime64, which is a subclass of numpy.ndarray and thus can be annotated as numpy.typing.NDArray[numpy.datetime64], especially as the return type of a function. Currently, Numpy type annotation doesn’t handle the shape of arrays, just the data type.

Set RF Gain in AM, FM, or SSB modes

Setting RF gain and squelch properly are vital to maximize receiver range (how far you can hear) while minimizing listening annoyance (background noise). It’s important to have a radio with NRC digital noise reduction, which can help reduce background noise and make listening more pleasant, especially in AM and SSB modes. CB radios with NRC are the same price or cheaper than 1990’s style CB radio without NRC, despite NRC being such a significant improvement to the listening experience. The technique to optimally adjust RF gain and squelch is distinct for each of the three main modes: AM, FM, and SSB (LSB/USB).

Separate and independent of RF Gain, Squelch, and NRC are the ANL (Automatic Noise Limiter) and NB (Noise Blanker) features. ANL and NB can often be “off” in base station or vehicle engine not running scenarios, which can reduce distortion of received signals, but with the penalty of more background noise. In scenarios with significant engine noise, ANL and NB can be turned on to reduce engine noise, but with the penalty of some possible distortion of received signals - often the benefit of ANL and/or NB outweighs the distortion penalty in noisy environments.

AM setting of RF gain and squelch

For AM, set the squelch to manual and open (you hear noise), then set the RF gain like:

  • knob: starting from fully clockwise RF Gain, turn counterclockwise just until the background noise noticeably drops.
  • menu: starting from “off” RF Gain, increase the number until the background noise noticeably drops.

This is the point where the RF gain is just enough to amplify the desired signal, but not so much that it amplifies the background noise and makes listening annoying. Now, turn up the squelch just till the noise stops - or turn on auto-squelch. This will maximize range on AM while helping reduce background noise. If there are DX signals via skip or a noisy area, adjust the RF gain and squelch accordingly to find the best balance between receiver range and noise reduction.

SSB setting of RF gain and squelch

On SSB modes (LSB / USB), across radio models from inexpensive to top-end, the squelch generally is too aggressive and will reduce receiver range if set to block all noise. It can be a better practice on SSB to not use the squelch if maximum range is desired, using only the RF gain to set noise to a low level. Experiment to find if squelch is useful for your scenario - perhaps if no conversation is happening, just close the squelch, and when a conversation starts, open the squelch to hear at better sensitivity.

FM setting of RF gain and squelch

On FM, the squelch is generally more effective at blocking noise than in AM or SSB modes. You can leave the RF gain as set in AM or SSB modes, or to maximize range, turn up the RF gain until the S-meter begins to indicate a signal, then adjust the squelch to just block the noise - or turn on auto-squelch. For DX communications, opening the squelch on FM will help just like on any FM radio - but of course this will let in noise between voice transmissions. Radios with NRC digital noise reduction will help reduce the noise of open squelch somewhat.