Scientific Computing

Matlab or GNU Octave can create lossless videos from image stacks or figure series by either:

• Save PNG image stack, then losslessly convert PNG stack to video
• Save from Matlab directly as lossless JPEG2000 video

This example saves lossless MJPEG2000 .mj2 video plots directly from Matlab VideoWriter.

v = VideoWriter('test.mj2', 'Archival');
v.LosslessCompression = true;
v.FrameRate = 5;  % arbitrary
open(v)

f = figure;
% or axes

pause(0.2)
% let plot initially draw

for i = 1:.05:3;

    line(0:.01:1, (0:.01:1).^i)
    % dummy plot sequence

    v.writeVideo(getframe(f))
end

close(v)

Play Motion JPEG 2000 .mj2 with FFmpeg from the command line (outside of Matlab):

ffplay movie.mj2

GNU Octave video package also has a “VideoWriter” function.

Consider the semi-automated script to install Debian to the eMMC of the Beaglebone Black.

This procedure requires a 4 GB micro SD card or larger, and assumes a Linux laptop. We will copy a Linux operating system image to eMMC from a micro SD card.

On laptop PC, download the Debian Beaglebone Black (BBB) image. For an old 2 GB Beaglebone, use the 2gb image.

Type lsblk, note which drives are listed, then insert the SD card into the laptop and type lsblk again–the new item is your SD card. We assume /dev/mmcblk0.

Extract image to SD card:

xz -cd BBB-*.img.xz > /dev/mmcblk0

Extraction takes about 5 to 20 minutes at ~ 5 MB/sec, writing uncompressed ~ 2 GB to the SD card.

[optional] monitor data writing to SD card with

iotop

Ensure writing has completed with

sync

Insert micro SD card into the (non-powered) BBB and then apply power. Beaglebone four onboard LEDs flash back and forth in a “cylon” or “knight rider” pattern. During this time, the micro SD card program is flashing the onboard eMMC. With an FTDI to USB adapter fit onto J1, the process can be monitored via the screen program. The automatic flashing to eMMC process should complete in about 10-20 minutes.

Once the BBB has shut down, REMOVE the micro SD card from the BBB. Reset the power or push the onboard POWER button next to the Ethernet jack.

Boot from eMMC

There is an SSH server running by default available through the mini-USB port.

Find the LAN IP address of the BBB (plug the BBB into the Ethernet). Assuming local network IP addresses 192.168.1.xxx, from a laptop:

nmap -p 22 192.168.1.* --open

This command lists SSH servers on the network, so do it once with the BBB unplugged from the network, then again with the BBB plugged in. If nmap isn’t available, the findssh program uses plain Python to find SSH or other servers.

Assuming BBB is at 192.168.0.5, from a laptop:

ssh debian@192.168.0.5

Notes

If timezone or locales issues try on Beaglebone:

dpkg-reconfigure tzdata
apt install locales
dpkg-reconfigure locales

Troubleshoot HDMI with parse-edid, obtained by:

apt install read-edid

Some programs simply cannot run without an X11 graphics server available. In other cases, testing a program requires X11 to be available, even though the windows aren’t visible. An example is creating Matplotlib figures in a continuous integration environment.

The X server virtual framebuffer (Xvfb) allows creating non-visible desktop windows without the computation burden of an full X11 graphics environment.

Xvfb is effective for most Linux systems without displays, particularly for headless or embedded systems including:

• Raspberry Pi
• Windows Subsystem for Linux
• continous integration servers
• virtual machines

apt install xvfb

Add to ~/.profile

Xvfb :99 &
export DISPLAY=:99

Upon opening a new Terminal or SSH connection, the X11 virtual framebuffer is available.

Test XVFB by running a graphics-only program, for example:

xmessage -center "Hello World"

If XVFB or X11 isn’t working, the message will be like

Error: Can't open display: :0.0

In general, computers using vPro remote access should not be directly exposed to the Internet. The computer’s software firewall doesn’t block vPro ports, which exist outside of the operating system. An external firewall is necessary to protect vPro remote access.

Remote firewall connections to Intel vPro machines can be made via SSH port forward to use Intel AMT KVM. Don’t open Intel AMT vPro ports ports to the public Internet. These ports are the minimum that we’ve observed are necessary to use Intel AMT for remote control, including remote power cycling.

To diagnose vPro remotely, first connect with the ports above forwarded to the laptop, then browse to https://localhost:16993. If the vPro VNC server is enabled, connect on Port 5900 with any VNC client over SSH tunnel. Use MeshCommander to connect via the ports above. Certificate instead of password is generally preferable.

Reference: Intel AMT network ports

With the recent release of iOS 13, we are seeing Apple iOS devices that suddenly won’t connect to any authenticated WiFi. The solution seems to be to do one more reboot of the iOS device and if needed, “forget” and re-add the desired WiFi system. Rebooting the WiFi router is generally NOT needed.

The FITS data file format is used in astronomical imagery. Two of the easiest ways to view FITS file image stacks with standalone programs are HDFView and NASA FV.

While FV has utilities and a UI more oriented to astronomical uses, HDFView is a generally useful tool to view HDF5, HDF4 and FITS files.

The Perl script “latexdiff” generates highlighted differences between two versions of a LaTeX document. This is required in submitting academic paper revisions.

• macOS: brew install latexdiff
• Linux / WSL: apt install latexdiff

Recursive latexdiff.py processes all .tex files in the directory. This is useful for very large projects like Ph.D. thesis or journal article.

A wide variety of programming languages are used by engineers and scientists. Tie them all together (C, C++, C#, Cuda, Fortran, etc.) in a platform-independent and simple way using CMake or Meson. These high-level build systems generate low-level build system backend files for Ninja (strongly recommended) or Make.

Assume a single-file C++ program that uses the Math library and Boost for flexible command-line input of numerous parameters. Turn on additional compiler warnings to help avoid common coding pitfalls. Consider an example CMakeLists.txt for a C++ and Fortran project, line by line.

Language(s) selection:

project(zakharov CXX)

Naming a project facilitates packaging and installation. CXX is required to enable the hooks for the language(s) you used. The most frequently used languages include

Languages that aren’t built into Cmake such as Pascal can be added via custom Cmake modules.

Compiler options:

if(CMAKE_Fortran_COMPILER_ID STREQUAL "GNU")
  add_compile_options(-mtune=native -Wall -Wextra -Wpedantic -fexceptions -Warray-bounds)
endif()

-mtune=native

use specialized optimizations for your particular CPU

-Wall -Wextra -Wpedantic -Warray-bounds

turn on warnings for common programming mistakes

-fexceptions

more detailed debug info with no speed penalty–enabled by default on Clang.

find_package(Boost REQUIRED COMPONENTS filesystem program_options)

We use Boost:

filesystem

directory manipulation

program-options

advanced command-line parsing

add_executable(zakh zakh.cpp)
target_link_libraries(zakh PRIVATE Boost::filesystem Boost::program_options)
target_compile_features(modules PRIVATE cxx_std_11)

This project requires C++11 features, so an old compiler not supporting C++11 will emit a configuration error.

zakh

the exe file that will be created on compile, run with ./zakh.

zakh.cpp

the files making up “zakh”

Compiling a simple project with CMake: It’s convenient to create a separate directory, typically build/ under your main code directory. Let’s say your main code directory is ~/code/zakharov, then do

# configure
cmake -B build/`

# build program
cmake --build build/ --parallel

# run program
./zakh

Let’s say you edit the code–rebuild and run by:

cmake --build build/ --parallel

./zakh

Normally you do not need to reconfigure CMake if just editing source code.

CMake alternatives include Meson.

Related:

• CMake changelog
• Cmake project directory structure

In Python, x**y is much faster than:

• pow(x, y)
• math.pow(x, y)
• numpy.power(x, y)

Julia is more than 5 times faster than Python at scalar exponentiation, while Go was in-between Python and Julia in performance.

Python

Benchmarking was the same for integer or float base or exponent.

Python testing done with:

• Python 3.7.4
• Ipython 7.8.0
• Numpy 1.16.5

** operator

The ** operator in Python also has the advantage of returning int if inputs are int and arithmetic result is integer.

10**(-3)
8.22 ns ± 0.0182 ns per loop (mean ± std. dev. of 7 runs, 100000000 loops each)

pow(10, -3)
227 ns ± 0.313 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

math.pow(10, -3)
252 ns ± 1.56 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

numpy.power(10., -3)
1.5 µs ± 2.91 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

Numpy is known in general to be slower at scalar operations than Python-native operators and Python built-in math. But of course Numpy is generally a lot faster and easier for N-dimensional array operations.

Julia

Julia 1.2.0 was likewise benchmarked under power/ for reference on the same computer.

First we installed Julia BenchmarkTools:

import Pkg
Pkg.add("BenchmarkTools")

The Julia wallclock time for exponentiation was the same for float and int as with Python.

3.399 nanoseconds

Go

Go 1.13.1 was benchmarked under power/:

go test -bench=Power

BenchmarkPower-12       33883672                31.8 ns/op

go benchmark reference

For scenarios where a function outputs a list, and that function is in a for loop or asyncio event loop, the final output will be a list of lists, like:

x = [[1,2,3], [4, 5, 6]]

This may be inconvenient for applications where a flattened list is required. The simplest and fastest way to flatten a list of lists in like:

import itertools

x = [[1,2,3], [4, 5, 6]]

x_flat = list(itertools.chain(*x))

which results in

[1, 2, 3, 4, 5, 6]