Scientific Computing

Best colormap for Matlab/Matplotlib plots

Much has been written on selecting best colormaps from among: sequential, divergingm and qualitative. Sequential colormaps are good for representing magnitude of data. How much flow, how much precipitation, how much weight, temperature, etc. Having a monotonic lightness factor is important for perceptual consistency. Non-linear lightness is used to emphasize certain ranges of data, perhaps where snow changes to ice or rain. Non-monotonic lightness can be used to emphasize different types of precipitation or phase changes, etc. Example sparse data plots with reversed sequential colormaps: colormap_white_min.py, colormap_white_min.m

Reversed sequential colormaps are useful for sparse data such as astronomical images or precipitation data where a lot of data is at or near zero relative to other data. The reversal leads to near-zero areas being white and higher intensities being darker. While any colormap can be reversed, typically sequential colormaps are used with/without reversal. Matplotlib colormaps are reversed by appending _r to the colormap name. For example:

cmap='cubehelix_r'

Matlab and GNU Octave colormaps are reversed by flipud() the colormap. Colormaps in .m code are represented as an (N,3) array, where N is the number of steps in the colormap (typically 64 or 256).

colormap(flipud(cubehelix()))

Matlab cubehelix.m is like Matplotlib.

Diverging colormaps are useful for positive or negative data where the sign is as important as the magnitude. For example, in/out flows, positive/negative charge. These colormaps are white near the zero point (which can be offset) and intensify as their absolute magnitude increases.

Qualitative colormaps emphasize difference between values, but without a particular sense of ordering. This can be useful for categories, say a histogram of salary vs. employee type.

Diagnosing SSH servers

This checklist helps rapidly diagnose SSH server issues, segmenting the problem into whether it’s network, firewall or SSH server configuration issues.

This checklist is OpenSSH + Debian/Ubuntu centric. Some systems (particularly embedded) may use iptables or nftables directly instead of high-level ufw.

Check if the SSH server is actually running. Errors in /etc/ssh/sshd_config can prevent the SSH server from running.

service ssh status

should show feedback including:

Active: active (running)

If not, try

service ssh restart

If restarting SSH server allows connections, be sure that after rebooting the server, the SSH server still works. This could help avoid a costly trip back to the site later.

From the laptop, error

Connection refused.

can simply mean the SSH server is not running.

Firewall: check port SSH server is supposed to be on in /etc/ssh/sshd_config. Look for Port 22 or whatever the SSH server is on. Although “security through obscurity” is not the goal of changing to a non-default SSH port, it can drastically reduce the amount of log flooding.

Check that firewall is open on the SSH server port.

ufw status

22/tcp ALLOW Anywhere

22/tcp (v6) ALLOW Anywhere (v6)

Check that packets are making it from the SSH client to the SSH server by on the server

tcpdump port 22 -n -Q inout

Specify the desired network interface as revealed by ip a with the tcpdump -i option. When the SSH client attempts to connect, the SSH server tcpdump should show packets coming in on the desired port with the client IP address also shown. If not, see if the network itself has a firewall that’s blocking your packets.

Logs: should see attempts to connect showing in /var/log/auth.log. Try

tail -f /var/log/auth.log

to see if new attempts show up.


Fail2ban might have the laptop IP address in “jail”. Check on the SSH server to see which IP addresses are banned:

fail2ban-client status sshd

Ubuntu Meltdown / Spectre patch

Ubuntu’s Meltdown/Spectre wiki as well as their Meltdown/Spectre press release indicate the versions of Ubuntu patched for this issue. Part of the general problem is servers that should be maintained in the cloud are instead running in a closet without maintenance.

A modern more robust approach:

  • Consider virtualization where possible, with remote staff dedicated to patches and maintenance
  • for long-term survivability of custom code assets, use OS-agnostic coding methods with modern build systems such as CMake.
  • Use continuous integration with unit tests and integration tests
  • common version control across code products

Registration tests are a first step on wrangling the dataflow lifecycle. Registration tests + continuous integration add meaningful testing across OS & compiler versions, monitoring edge cases that break. Driving toward unit tests incrementally as warranted allow minimal upfront expenditure.

Julia IDE

A popular IDE for Julia (syntax highlighting, linting) is via the Julia plugin for Visual Studio code.

Radio Mobile Deluxe WINE

Radio Mobile Deluxe is free but not open source radio propagation Windows software that runs on Linux well via WINE. Installing Radio Mobile Deluxe is straightforward:

Download rmwcore.zip and extract to ~/rmw. Download English (or other) language code e.g. rmw1166eng.zip and extract to ~/rmw Install Visual Basic 6 runtime:

winetricks vb6run

Make a script ~/rmw.sh containing

#!/bin/sh

wine ~/rmw/rmweng.exe

Then:

chmod +x ~/rmw.sh

Create a new app icon by creating a file ~/.local/share/applications/rmw.desktop with contents:

#!/usr/bin/env xdg-open
[Desktop Entry]
Type=Application
Exec=wine ~/rmw/rmweng.exe
Name=Radio Mobile Deluxe

Use the Radio Mobile Deluxe menu shortcut or in Terminal

~/rmw.sh

Antenna Patterns

Radio Mobile Deluxe .ant antenna pattern files require Windows/DOS \r\n CRLF line endings even when run under WINE. Convert Unix LF line ending files with

unix2dos myfile.txt

When using Python to write text files in Linux, by default the line endings will be \n LF line endings. Force Python to write Windows/DOS CRLF from Linux when using functions with options such as:

numpy.savetxt(newline='\r\n')

That works across operating systems.


Radio Mobile Deluxe through version 11.6 does not reload modified .ant files. You must close/reopen the entire RMD program to reload a modified antenna pattern file.


err:module:import_dll Library MSVBVM60.DLL (which is needed by L"Z:\rmw\rmweng.exe") not found

Fix:

winetricks vb6run

Alternative open source wireless propagation software:

  • Splat RF is also great for radio coverage prediction outdoors.
  • PyLayers is for indoor site-specific radio coverage prediction, also suitable for UWB.

Fix vapor locked gasoline pump

At gasoline pumps in 5°F / -15°C weather, everyone was complaining that the pumps were unusably slow, just a few hundredths of a gallon per second. After everyone stopped pumping for 30 seconds, the system escaped vapor lock and pumping resumed as normal. Vapor lock implies high volatility gasoline has flashed to vapor somewhere in the suction pumping gas station system. Stopping pumping entirely allowed that portion of the system to cool, so the gasoline went back to liquid.

About 45 seconds into the video, we all stopped pumping, and when the video resumes, everyone is pumping normally. If at a fueling station and all the pumps are extremely slow, suggest that everyone stop for 30 seconds and then try again.

EPA regulations on summer fuel vapor pressure

Use wired Wifi APs instead of mesh network

Worthwhile mesh networked Wifi APs (a.k.a. “tri-band”) use a separate simultaneous channel for the backhaul.

Mesh networked Wifi AP should:

  • use two simultaneous 5 GHz channels (one for backhaul/mesh, one for users)
  • even better, use 60 GHz backhaul
  • have a graphical screen/web interface that encourages users to reduce number of hops by tweaking AP location

In general Wifi APs should:

  • be placed in inner rooms or the inner side of rooms away from windows.
  • use wired, MoCA or powerline Ethernet connections to each AP.

Wifi APs should be placed (from most preferable to least preferable)

  • in each high usage room (den, kitchen, bedroom), and place a Wifi AP at least every other room for large homes
  • in smaller homes, one Wifi AP per home level, near the center of each level

5 GHz on all APs, enable 2.4 GHz only on ONE most central AP to account for IoT devices without 5 GHz.

Wifi AP Performance Notes

  • We should always strive for wired Ethernet connections or MoCA to each AP as today’s best HomePlug and G.hn MIMO powerline Ethernet are much closer to 100 Mbps throughput instead of the 1000 Mbps displayed prominently on the package.
  • Keep in mind the latency introduced by mesh networking and its impact on live streaming video.
  • Wifi repeaters are even worse w.r.t. latency.

Related: WiFi channel, SSID, power best practices

FFmpeg concatenate video files

FFmpeg can losslessly (without reencoding) concatenate videos, even in different container formats, as long as the codec is the same. If the codec is different between files (H.264 vs. FFV1) then you would instead need to reencode.

This method can be used for YouTube video uploads, where you want to join small video files into one big file without bothering with a video editor.

Concatenate same codec videos

This example assumes you have MP4 videos to concatenate. If you’re on Windows, you will need Windows Subsytem for Linux.

Create a specially formatted file listing files to convert.

for f in ./*.mp4; do echo "file '$f'" >> flist.txt; done

Concatenate:

ffmpeg -f concat -safe 0 -i flist.txt -c copy out.mp4

This produces out.mp4, which will be close to the sum of the file sizes. It’s not precisely the same number of bytes as the sum of the files, perhaps due to timestamping and metadata differences.

Notes

  • -pattern_type glob -i '*.mp4' does NOT work because that’s for the input filter image2 ONLY!
  • FFmpeg concatenate

Upload animated GIF to YouTube via FFmpeg

Losslessly convert animated GIF to AVI for high-quality YouTube upload:

ffmpeg -i my.gif -c:v huffyuv my.avi

Working

Currently, lossless HuffYUV works for YouTube uploads. HuffYUV is a primitive lossless compression algorithm, but you don’t want YouTube to doubly-lossy encode a lossy input video.

NOT working

  • ProRes: Video codec isn’t recognized by YouTube
  • FFV1: Video is often heavily clipped horizontally, half the video panel is missing