Ensure required packages present for pip install on offline systems

Offline (non-Internet-connected) systems may experience failures on installing Python packages. The package authors may not have CI setup for an offline test case, so they don’t realize it’s an issue. In general, Python packages can use setup.py, setup.cfg and pyproject.toml more effectively in a Python-standard way to overcome these issues.

Install package before setup.py

Instead of telling users to manually install a package such as Numpy that’s required before fully running setup.py, use pyproject.toml instead of setuptools setup_requires. This is because currently, setuptools assumes the computer will be internet-connected and even if the package is already installed, setup.py will fail on an offline system.

good example (most reliable and standard)

To ensure a package like Numpy is installed before setup.py fully runs, for example where f2py is used, have a pyproject.toml file including:

[build-system]
requires = ["setuptools", "wheel", "numpy"]

This will auto-install the prereqs before setup.py runs.

When including this pyproject.toml parameter, do not omit “setuptools”, “wheel” or the package may fail to install. PEP517 assumes unless told otherwise:

[build-system]
requires = ["setuptools", "wheel"]

such as by omitting them in a pyproject.toml file.

bad example (don’t do this)

These may fail on non-internet-connected systems, even if Numpy is already installed:

setup.py

setuptools.setup(..., setup_requires=['numpy',])

or

setup.cfg

[options]
setup_requires =
  numpy

pip install downloaded .whl file

Pip is a widely-used, complex Python package installer program with a lot of legacy baggage. The Python Software Foundation recognizes the critical need to update Pip, putting $116K to sponsor a senior dev to modernize Pip.

Sometimes, pip install fails to realize a .whl binary wheel is available. Thus pip tries to download and install a package from source code. In the case of a large package like SciPy, OpenCV or Pillow on an embedded system like the Raspberry Pi Zero, it could take hours or even days to compile, probably failing numerous times due to missing prerequisite binary libraries.

Manual install wheel

A workaround to Pip not automatically finding the desired .whl binary wheel is to download it manually and install the .whl directly. The binary wheels are often available at PyPi from the package download page, for example SciPy. For embedded systems such as Raspberry Pi, there may be non-PyPi sites such as PiWheels.

Download the file, then pip install from the file like:

https://www.piwheels.org/simple/scipy/scipy-1.3.2-cp37-cp37m-linux_armv7l.whl

Troubleshooting

If the wheel binary is not compatible with the system, it will fail to import or run. In that case, simply pip uninstall my_package_name and try something else.

Homebrew binary bottle download

Downloading binary Homebrew bottles without installing Homebrew can be useful to check the location of the bottle contents. This is useful when developing Meson native-files or CMake Find*.cmake modules.

Since 2015, Homebrew distributes bottles from Bintray. For example, HDF5 bottle may be inspected by:

tar --list -f hdf5-1.10.5_1.catalina.bottle.tar.gz

No Homebrew install is necessary for inspection, but actually using the libraries and binaries is of course best done by installing Homebrew.

MyPy Python type checking single file or files

Usually it’s desired to check Python type annotations for an entire project. However, when introducing type checking to a large Python project, it can be beneficial to introduce type checking gradually, to avoid massive refactoring all at once. You will almost certainly find old code bugs due to type checking that will require refactoring, which introduces risk. Mitigate this risk by introducing type checking gradually.

MyPy single file options

MyPy is a strongly recommended type checker, though others exist. We typically use a per-project mypy.ini to make appropriate MyPy default behavior. To tell MyPy to only check certain files, use the MyPy –follow-imports= option like:

mypy --follow-imports=skip myproj/foo.py myproj/bar.py

Only certain directories can be checked like:

mypy --follow-imports=skip myproj/sub/ myproj/sub2

Once the type checks pass via mypy --follow-imports=skip, we recommend trying

mypy --follow-imports=silent

to improve robustness of the type check for those files / directories.

Install PGI free Fortran compiler

Related: Install Flang LLVM Fortran compiler

The PGI Community Edition compilers are a set of no-cost, non-open-source, high performance compilers for:

  • C: pgcc
  • C++: pgc++ (C++17, Mac / Linux only)
  • Fortran: pgfortran (Fortran 2008) including CUDA Fortran.

Install

Note: if you have an older PGI compiler installed, uninstall that first to avoid license server issues–even when using the no-cost Community Edition.

Download and install PGI Community Edition:

  • Linux: sudo not required, but must be on symbolic-link-aware drive (not ExFAT)
  • MacOS: requires Xcode, sudo not required, but must be on symbolic-link-aware drive (not ExFAT)
  • Windows: Requires Visual Studio

PGI compiler is over 2 GB download. CUDA install is optional.

Linux

We assume PGI was installed under “~/.local/pgi” – add to ~/.bashrc like:

export PATH="$PATH:$HOME/.local/pgi/linux86-64/2019/bin/"

Open a new terminal to use PGI pgfortran pgcc pgc++

MacOS

On MacOS in general we recommend the Bash shell, enabled by one-time setting

chsh -s /bin/bash

MacOS requires Xcode to use PGI.

We assume PGI was installed under “~/.local/pgi” – add to ~/.bashrc like:

export PATH="$PATH:$HOME/.local/pgi/linux86-64/2019/bin/"

Open a new terminal to use PGI pgfortran pgcc pgc++

Windows

use PGI command prompt–you may need to do

set FC=pgfortran
set CC=pgcc

Note that there is no pgc++ on Windows.

PGI requires Visual Studio. If you upgrade Visual Studio (even a minor update) after installing PGI, you may get errors on PGI compiling like

pgfortran my.f90

pgfortran-Error-Required tool link was not found

The easiest solution is to just reinstall the PGI compilers.

Usage

Typically we invoke compilers from a build system such as Meson or CMake.

Meson

Meson ≥ 0.50 supports PGI and Flang compilers.

project('demo','fortran')

fc = meson.get_compiler('fortran')
if fc.get_id() == 'pgi'
  message('PGI compiler')
endif

CMake

CMake ≥ 3.10 distinguishes between PGI and Flang compilers. Set compiler-specific options in CMakeLists.txt for various Fortran compilers like:

cmake_minimum_required (VERSION 3.10)
project(myproj Fortran)

if(CMAKE_Fortran_COMPILER_ID STREQUAL PGI)
  add_compile_options(-Mdclchk)
endif()

select the PGI compilers at CMake configure step:

FC=pgfortran CC=pgcc CXX=pgc++ cmake -B build

PGI Debugger

PGI 19.7 deprecated PGI debugger – it will be removed after 2019. PGI Java-based pgdbg graphical debugger was for Fortran, C and C++, including the no-cost Community Edition. The program to be debugged needs compile options -g -O0 to provide debugging information.

Example with hello.f90:

  1. pgfortran -g hello.f90
  2. pgdbg a.out opened the graphical Fortran debugger.

If you don’t see code in the upper left of the graphical PGI debugger for your program, be sure you compiled the executable with -g -O0 options.

Check if Python interpreter is 32 or 64 bit

Like most other programs, 32-bit Python can be run on 64-bit OS. When needing to precisely detect if the operating system is 64-bit or 32-bit in a cross-platform way, one needs to check if Python itself is 32-bit to avoid falsely detecting that the operating system is 32 bit when the OS is actually 64 bit.

Note: This check is just for the Python interpreter. If you’re working on detecting the operating system parameters in a cross-platform-robust way, several further checks are necessary in general.

CLI example

python -c "import sys; print(sys.maxsize > 2**32)"
  • 32-bit Python will print False
  • 64-bit Python will print True

Functional

import sys

def is_64bit() -> bool:
    return sys.maxsize > 2**32

Notes

reference

Five free C C++ Fortran compiler families

Dozens of “free” compilers exist for C, C++ and Fortran. However, only five compiler families available at no charge support the modern features of these popular compiled languages.

GCC

GCC has broad support of modern standards on a very wide range of computing platforms. GCC’s downside in some cases can be slower runtime performance than compilers having less broad language and platform support.

LLVM

LLVM Clang and Flang have significant industry support, including from Nvidia, and are known for high performance, but somewhat less language feature support and less broad platform support than GCC.

PGI

PGI Community Edition is free to use, and has performance comparable to Intel compilers in many cases. PGI ≥ 19.10 should be used for modern Fortran. PGI supports CUDA Fortran.

Intel

Intel compilers are a paid product in general. For non-commercial purposes, including:

  • educators
  • students
  • open-source projects

the Intel compilers are available at no charge. The Intel performance libraries like MKL, IPP, TBB and more are available at no cost with more liberal use terms.

IBM XL

IBM XL compilers are currently for POWER CPUs only e.g. ppc64le. IBM XL compilers do not work with a typical x86-based computer. If you have a $3000 Raptor IBM POWER9 desktop, then IBM XL may be for you.

The IBM XL compilers are high-performance compilers that have a free community edition. IBM XL Fortran has wide support for Fortran 2008.

Git global recursive directory ignore

Many IDEs create per-project cache directories with metadata relevant to the IDE configuration for that project. Instead of editing the .gitignore file for each repository, ignore directories in Git system-wide for all repos as follows, for any operating system.

System-wide .gitignore

Tell Git where the global ~/.gitignore file is:

git config --global core.excludesfile ~/.gitignore

Create / edit the ~/.gitignore file to include things like:

.vs/
.vscode/
.idea/

.mypy_cache/
.DS_Store

*.pyc

Turn off Connected Standby on Windows 10

To avoid connection failures requiring reboot of a Windows 10 PC upon resuming from sleep/standby, consider disabling Windows network (WiFi) connectivity during sleep. Only certain PCs are capable of having network connectivity during sleep. The option to disable WiFi during sleep is only present if the laptop supports it.

Windows Wifi power setting is under the Control Panel → System → Power & Sleep. If present, uncheck under “Wi-Fi” the “stay connected…sleep” options.

Check if laptop supports Connected Standby

From Command Prompt, type:

powercfg /a

if the response includes

Standby (S0 Low Power Idle) Network Disconnected

then the PC is set to disconnect from WiFi on sleep.

Notes

Advantages of disabling Windows Connected Standby:

  • save battery by not having your computer connected to Wifi while sleeping.
  • not connecting to possibly rogue AP as you travel about, or being subject to tracking.
  • some hardware (e.g. Microsoft Surface Pro) often fails to connect to Wifi on wakeup until manually going to Device Manager and disable/enable cycling the Wifi adapter.