Scientific Computing

CI examples

Continuous Integration services run user-defined self-checks on code on each “git push”. GitHub Actions is often the best place to start with CI. CI is one part of the DevOps lifecycle.

These services have quotas even for public Git repos:

At the time of this writing, these CI didn’t have usage quotas for public repos. They are constrained by the number of concurrent jobs.

We have amassed numerous CI examples across programming languages, including

Microsoft Surface lifecycle

Microsoft Surface computers are generally a good device to consider for education, work and personal computing. Microsoft has published a firmware update lifecycle schedule for all Surface devices. While one would assume after four years or so of firmware updates the bugs would mostly be worked out, there is the possibility of an unpatched firmware vulnerability as has happened across OEMs.

IPython no confirm on exit

IPython “confirm on exit” is not always desired. To disable confirm on exit, from Terminal (not in Python) type:

ipython profile create

This will print out the locate of the IPython config file, which is like “~/.ipython/profile_default/ipython_config.py”. Open this file in a text editor, and uncomment the line:

c.TerminalInteractiveShell.confirm_exit = False

Upon restarting IPython, the usual Ctrl+D immediately exits without confirmation.

GitHub Actions quick start

GitHub Actions is among the best no-cost CI and is relatively easy to use. Here are a couple example GitHub Actions templates we use across many projects. Each workflow is stored in the project under directory “.github/workflows”. For clarity, usually we use a filename like “.github/workflows/ci.yml”.

This Python template tests on Linux, macOS and Windows. Always enclose version numbers in quotes to avoid YAML interpretation issues.

The optional “Unix FC var” step is an example of setting an environment variable dynamically–here for a Fortran compiler.

name: ci

on:
  push:
    paths:
      - "**.py"
      - .github/workflows/ci.yml


jobs:

  core:
    strategy:
      matrix:
        os: [ubuntu-latest, windows-latest, macos-latest]

    runs-on: ${{ matrix.os}}

    steps:
    - uses: actions/checkout

    - uses: actions/setup-python
      with:
        python-version: '3.x'

    - run: pip install .
    - run: pip install flake8 mypy pytest

    - run: flake8
    - run: mypy

    - name: Unix FC var
      if: runner.os != 'Windows'
      run: echo "FC=gfortran-14" >> $GITHUB_ENV

    - run: pytest

Python vs. Matlab for command line

The Matlab -batch command line option makes running Matlab code from the command line more reliable. However, Matlab is not really designed to be used extensively by end users this way. Matlab does not have a command-line parser like many other languages (scripted and compiled) do.

Python has the argparse module that we generally find sufficient. In Fortran, using the get_command_argument() with select case can make a straightforward command line parser without resorting to external packages.

Perhaps a reason for Matlab not having these features is the startup time of Matlab. While Python can complete a small program in 100 ms from the shell, Matlab takes 10-30 seconds or so depending on the number of toolboxes and the speed of the computer. GNU Octave starts up nearly as quickly as Python when Octave toolboxes are not auto-loaded. It would be nice if Matlab someday will make an actual command-line interface with fast startup time.

CI select OS version

Many CI services use Ubuntu Linux images. The default Linux / Ubuntu version is typically an older version. It can be convenient to switch CI jobs to a newer Linux version to get more recent libraries. Here are examples of switching Linux version for a few popular CI services.

GitHub Actions default Ubuntu version is controlled with .github/workflows/*.yml file(s).

jobs:

  linux:
    runs-on: ubuntu-<version_number>

Profile and speedup Matlab functions

Code profiling and micro-benchmarks can be useful to discover hot spots in code that might be targets for improving efficiency and overall execution time. Matlab has long had an effective profiler built in from the factory.

Commercial and free software often prioritized quality and correctness over speed as part of a good quality software product. There are cases where the Matlab factory functions are simply not optimized due to development time constraints. Yair Altman has expertise with the undocumented scripts inside Matlab that many Matlab users rely on. He has written a three-part series on profiling and optimizing Matlab factory code, which I find to be worthwhile reading even for general Matlab code optimization.

Speeding up builtin Matlab functions:

macOS Homebrew without sudo / admin

Homebrew is a popular framework for quickly installing development tools on macOS, including Gfortran. macOS cloud services such as MacInCloud may say they provide Homebrew, but you may not be able to install packages without a sudo/admin account. To install Homebrew without sudo/admin in the user home directory for cloud or physical hardware, follow these steps:

mkdir ~/homebrew && curl -L https://github.com/Homebrew/brew/tarball/master | tar xz --strip 1 -C ~/homebrew

If one truly doesn’t have sudo / admin access as typical with a managed (less-expensive) cloud macOS plan, and if Xcode is not the appropriate version, GCC may compile from source, which can take tens of minutes. This may occur when doing

brew install gcc

Reference: Homebrew without sudo script

Matlab Compiler tbb.dll oneTBB

Running compiled M-scripts using Matlab Compiler deploytool may result in warning messages about tbb.dll. Updating the Threaded Building Blocks library may help. Also consider updating the Matlab version if applicable.

Download and install latest Intel oneTBB. If using a command prompt, make oneTBB available by command tbbvars. Recompile with Matlab deploytool.

Default MyPy type hint checks

MyPy is a Python type annotation checker. MyPy recursively checks all files in a Python project by typing:

mypy ~/myproject

We typically use the PyPI MyPy instead of conda to have the most recent MyPy version:

pip install mypy

It’s often useful to have a per-project MyPy configuration file to avoid excessive command line options. Configure MyPy defaults in pyproject.toml:

[tool.mypy]
files = ["src", "Examples"]
ignore_missing_imports = true
strict_optional = false
show_column_numbers = true

Where “files” is set appropriately for the project. Making a per-project files is strongly recommended to ensure files aren’t missed in the type check. One can make a system-wide ~/.mypy.ini, that is overridden by the per-project pyproject.toml.

Enhanced mypy usage