Scientific Computing

WSJT-X operating tips

The WSJT-X program enables popular digital modes like FT8 and FT4 among several other modes. WSJT-X is generally easier to use than FLdigi, another popular digital program that handles a wide range of other ham radio data modes. WSJT-X decodes multiple signals at once, and discriminates between overlapping signals. WSJT-X includes multiple high performance digital modes, each tuned for challenging propagation conditions from LF to microwave.

Control transmitter RF output power with the sound card output level, NOT with radio RF power control. Optimize transmitter RF cleanliness by:

  1. set the RF power control to well above the intended transmit power level (say 6 dB to 10 dB higher)
  2. set microphone gain to about midrange
  3. set computer audio volume to precisely set output RF transmit power

If the radio transmitter is driven into ALC by using RF power control to limit transmitter level, the radio will splatter, interfering with adjacent frequencies. While some operators use QRP ≤ 5 watt transmit power for WSJT-X supported modes, many others use 25 .. 100 watts. This accounts for the discrepancy one might experience in a QSO between the received signal strength by each operator. Some use a very minimal amount of RF power for a “kilometers per watt” challenge.

When using CAT control for PTT, RFI (for example, common mode RF interference) can cause the radio to fall out of transmit, even with very low RF transmit power like 100 milliwatts. With the compromise HF antennas enabled by these efficient digital modes, one must be just as mindful of proper grounding and RFI as with a 100 watt SSB or CW station. Diagnosing this problem can be done by setting the transmitter power to as low as measurable, and seeing if the radio still drops out of transmit. If so, then the problem is likely RFI or common mode RF on the USB control cables.

Older radios may have too-narrow receive filters to capture the whole audio passband used by modern data-friendly transceivers. If the radio has “IF shift” or “pass band tuning”, it might receive from 400-2800 Hz. The “split” transmit feature of WSJT-X shifts the radio RF transmit frequency and audio frequency to optimize the transmit audio passband to minimize transmitted audio harmonics.

A radio that receives on multiple HF bands simultaneously greatly increases the data gathering capability for ionospheric studies. SDRs oriented toward “traditional” ham radio use typically receive one or two RF bands at once. Simultaneous multi-band reception can be accomplished with a bank of SDR receivers or a broadband direct-sampling SDR.

Matlab + Octave unit tests quick-start

Matlab unit test framework is well-established and xUnit-like. The Matlab unit test framework is incompatible with the GNU Octave BIST unit testing framework. The Matlab unit test syntax is completely different from the Octave unit test syntax. Nonetheless, a common ground between Matlab and GNU Octave can be found in the easy to use Matlab script-based tests. A Matlab script-based test can be run as an ordinary script with GNU Octave.

A key limitation of Matlab script-based tests is they can’t use the Qualifiable functions (verify*, assert*, assume*). This also means you cannot mark a test as skipped or Incomplete (using assume*) with script-based tests. Basic tests can work without those functions, but more advanced tests suites do strongly benefit from those functions.

Matlab or Octave can run a script-based test as a plain script. The downsides of running a script-based test as a plain script are:

  1. there is no decorated TestResult
  2. the first failure ends the whole test

It’s best to put the script-based test scripts in the same directory as the code they’re testing to avoid Matlab path issues.

When using Matlab on this Octave-compatible script-based test, a richer result comes from using Matlab-only runtests(). Matlab runtests() will search all subdirectories under the current working directory and run all scripts with case-insensitive “test” in the filename. For example, a script testing HDF5 function of a program might be named TestHDF5.m or HDF5test.m. This is similar to PyTest filename-filtering.

Denote each test in the script-based test file with a double percent sign like:

% test_example.m
% setup code goes up top, as each test's variables are isolated from each other
% when using Matlab runtests()

A = magic(2);

%% test_dummy
B = A*2;  % only visible to this test, when using Matlab runtests()
assert(isequal(B, A*2), 'dummy example')

%% test_four
C = A*4; % note that "B" is not visible under runtests()
assert(isequal(C, A*4), 'dummy four')

Run Matlab runtests() from CI command like:

matlab -batch "assertSuccess(runtests)"

Just doing matlab -batch runtests will NOT fail in CI, even with failing test.

Octave runtests('.') only runs tests in specially formatted comments that Matlab ignores. For projects that need to have tests supporting Octave and Matlab, we generally recommend writing Matlab script-based tests, and manually running each file from Octave.

Linking HDF5 with CMake for C, C++ and Fortran

CMake links HDF5 into the C, C++, or Fortran program with just two lines in CMakeLists.txt. If experiencing trouble finding HDF5 with CMake, try FindHDF5.cmake, which is more up to date than the FindHDF5.cmake included with CMake. An example CMake for writing network data to HDF5 in C: CMakeLists.txt.

We show an example for C and another example for Fortran. “HL” refers to the high-level HDF5 interface that is more convenient and thus commonly used.

Note: if terminal has the Conda environment loaded and you keep getting the Conda HDF5 library, do first:

conda deactivate

before running the CMake configure commadn.

project(myproj LANGUAGES C)

find_package(HDF5 REQUIRED COMPONENTS C HL)

add_executable(myprog myprog.c)
target_link_libraries(myprog PRIVATE HDF5::HDF5)
project(myproj LANGUAGES Fortran)

find_package(HDF5 REQUIRED COMPONENTS Fortran HL)

add_executable(myprog myprog.f90)
target_link_libraries(myprog PRIVATE HDF5::HDF5)

HDF5 C example

The Fortran HDF5 syntax is quite similar.

#include "hdf5.h"

int main(void) {

   hid_t       file_id, dataset_id,dataspace_id; /* identifiers */
   herr_t      status;
   int         i, j, dset_data[4][6], read_data[4][6];
  hsize_t     dims[2];

   /* Initialize the dataset. */
   for (i = 0; i < 4; i++)
      for (j = 0; j < 6; j++)
         dset_data[i][j] = i * 6 + j + 1;

   /* Create a new file using default properties. */
   file_id = H5Fcreate("test.h5", H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT);

   /* Create the data space for the dataset. */
   dims[0] = 4;
   dims[1] = 6;
   dataspace_id = H5Screate_simple(2, dims, NULL);

   /* Create the dataset. */
   dataset_id = H5Dcreate2(file_id, "/dset", H5T_STD_I32BE, dataspace_id,
                          H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT);

   /* Write the dataset. */
   status = H5Dwrite(dataset_id, H5T_NATIVE_INT, H5S_ALL, H5S_ALL, H5P_DEFAULT,
                     dset_data);

   /* End access to the dataset and release resources used by it. */
   status = H5Dclose(dataset_id);

//------------------------------------------------------

   /* Open an existing dataset. */
   dataset_id = H5Dopen2(file_id, "/dset", H5P_DEFAULT);


   status = H5Dread(dataset_id, H5T_NATIVE_INT, H5S_ALL, H5S_ALL, H5P_DEFAULT,
                    read_data);

   for (i = 0; i < 4; i++)
      for (j = 0; j < 6; j++)
        printf("%d ",read_data[i][j]); // 1-24

   /* Close the dataset. */
   status = H5Dclose(dataset_id);

   /* Close the file. */
   status = H5Fclose(file_id);

   return 0;
}

HDF5 compiler script

As an alternative to CMake, HDF5 compiler script h5cc links HDF5 and necessary libraries:

h5cc myprog.c func.c -lm
  • h5cc: C
  • h5c++: C++
  • h5fc: Fortran

HD Radio interference to license-free FM

In urban and extended suburban areas typical of southern California and the northeast United States, it can be difficult to find a clear broadcast FM (87.9-107.9 MHz) channel to use for license-free transmissions. As 2020 drove many communities to outdoor activities, many organizations used license-free FM transmitters to share music, lectures and entertainment a large audience locally in their automobiles. An issue we’ve observed is poor audio quality and erratic coverage of the license-free FM system due to sideband spillover of adjacent HD Radio broadcasts. With HD Radio, the first adjacent channel spillover power is hundreds to thousands of times higher on average than with an analog FM signal alone. The usual sound of HD Radio interference is simply a hissing noise that is difficult to distinguish from ordinary noise when a station isn’t present.

The typical community member new to license-free FM would have difficulty distinguishing HD Radio first adjacent channel interference from simply having poor range. Without special equipment beyond an automobile or handheld FM radio, it’s difficult for the inexperienced person to detect the interference. A $20 RTL-SDR USB radio stick would give a spectrum analyzer view showing the HD Radio sidebands spilling within 100 kHz of the license-free carrier frequency. This energy will cover over the license-free FM broadcast. Without such equipment, look in the HD Radio database to see if the FM stations on the channel above and below have HD Radio. If so, pick a different clear license-free FM channel, if the first adjacent stations are clearly audible on your radio when on their channels.

Example

Suppose a first adjacent HD Radio station has a carrier signal strength at the radio of about -70 dBm with a typical car radio antenna. -70 dBm is typically more than adequate for clear-sounding broadcast FM audio when in an outdoor environment where there’s not excessive interference. Take a conservative and simple estimation that the HD Radio first adjacent has a -20 dBc HD Radio injection level, and that 50% of that power (-23 dBc) falls within the license-free receiver passband. This means roughly -90 dBm is on-channel of the license-free FM station. The area to be covered was about 150 meters from a theoretical central location for the FM transmitter. This would be well in excess of that expected for a legal license-free FM transmitter. For us the license-free FM program audio was only detectable over about 80% of the event, when we were probably within about 100 meters of the transmitter. However the audio quality was only usable for about 50% of the park.

We didn’t attempt to quantify this further, but this poor radio coverage would be expected from the use of a first adjacent channel to an HD Radio station, and trying to cover too large an area with a license-free FM transmitter. Once might suppose the FM transmitter used might have more than legal FM transmit power. Even in that case, one should not use a first adjacent channel to HD Radio, as this can drastically reduce coverage. Again, a key metric is if the first adjacent HD Radio channel (say 88.9MHz) is clearly audible when tuned to that channel (88.9), then trying to broadcast license-free on a first adjacent channel (88.7 and 89.1) will be a futile effort.

Illegal broadcasts also seek clear FM frequencies that are naturally not in any official database. Before printing promotional materials for your event, it’s important to verify with a local FM receiver than the planned channel is actually clear. Also consider that another operation may choose the same FM frequency near your event. It may be better to simply instruct attendees to look for big signs near the entrance(s) of your event so that the FM frequency can be quickly changed if interference is experienced. Be sure someone on-site knows how to change the FM transmitter frequency.

Reference: FM Receiver measured performance

CMake compiler feature detect

For nearly all CMake projects we choose to detect compiler features rather than make brittle, difficult to maintain nested if/elseif based on compiler vendor and version. There are still cases where if() statements to work around known compiler bugs are necessary, but where we can, we use compiler feature checks instead.

An example CMake project requires C11 variable length arrays. The CMakeLists.txt would look like:

include(CheckSourceCompiles)

set(CMAKE_C_STANDARD 11)
# so that check_source_compiles works

set(CMAKE_TRY_COMPILE_TARGET_TYPE STATIC_LIBRARY)
# save link time, only compile is needed

check_source_compiles(C "int main(void){
for (int i = 1; i < 5; i++){
  int a[i];
}
return 0;
}"
c11vla)

if(NOT c11vla)
 # fatal error or disable feature using C11 VLA
endif()

add_executable(c11demo demo.c)

CI examples

Continuous Integration services run user-defined self-checks on code on each “git push”. GitHub Actions is often the best place to start with CI. CI is one part of the DevOps lifecycle.

These services have quotas even for public Git repos:

At the time of this writing, these CI didn’t have usage quotas for public repos. They are constrained by the number of concurrent jobs.

We have amassed numerous CI examples across programming languages, including

Microsoft Surface lifecycle

Microsoft Surface computers are generally a good device to consider for education, work and personal computing. Microsoft has published a firmware update lifecycle schedule for all Surface devices. While one would assume after four years or so of firmware updates the bugs would mostly be worked out, there is the possibility of an unpatched firmware vulnerability as has happened across OEMs.

IPython no confirm on exit

IPython “confirm on exit” is not always desired. To disable confirm on exit, from Terminal (not in Python) type:

ipython profile create

This will print out the locate of the IPython config file, which is like “~/.ipython/profile_default/ipython_config.py”. Open this file in a text editor, and uncomment the line:

c.TerminalInteractiveShell.confirm_exit = False

Upon restarting IPython, the usual Ctrl+D immediately exits without confirmation.

GitHub Actions quick start

GitHub Actions is among the best no-cost CI and is relatively easy to use. Here are a couple example GitHub Actions templates we use across many projects. Each workflow is stored in the project under directory “.github/workflows”. For clarity, usually we use a filename like “.github/workflows/ci.yml”.

This Python template tests on Linux, macOS and Windows. Always enclose version numbers in quotes to avoid YAML interpretation issues.

The optional “Unix FC var” step is an example of setting an environment variable dynamically–here for a Fortran compiler.

name: ci

on:
  push:
    paths:
      - "**.py"
      - .github/workflows/ci.yml


jobs:

  core:
    strategy:
      matrix:
        os: [ubuntu-latest, windows-latest, macos-latest]

    runs-on: ${{ matrix.os}}

    steps:
    - uses: actions/checkout

    - uses: actions/setup-python
      with:
        python-version: '3.x'

    - run: pip install .
    - run: pip install flake8 mypy pytest

    - run: flake8
    - run: mypy

    - name: Unix FC var
      if: runner.os != 'Windows'
      run: echo "FC=gfortran-14" >> $GITHUB_ENV

    - run: pytest