If the wget HSTS database file permissions get changed, an error may occur on using wget like:
Will not apply HSTS. The HSTS database must be a regular and non-world-writable file.
ERROR: could not open HSTS store at '~/.wget-hsts'. HSTS will be disabled.
HSTS
can enhance security, so normally we’d like to have HSTS working.
Fix
Make the .wget-hsts file have normal file permissions by:
Intel MKL has Scalapack, which is installed when you select “Cluster Support” for C and/or Fortran from the Intel Compiler / Intel MKL installer.
MKL Scalapack library & include files are under the MKLROOT environment variable for Linux and Windows, like $MKLROOT/{include,lib/intel64}
Several more link flags (including for MPI) are needed to make Intel MKL Scalapack work.
The exact link flags and compile flags for a particular system are obtained from Intel
Link Line Advisor.
Examples
MKL
The Intel MKL Scalapack examples are compiled by:
cd $MKLROOT/examples/
tar xf examples_cluster_f.tgz
cd scalapackf
make sointel64 mpi=openmpi compiler=gnu mpidir=/usr/
The resulting “exe” files are under scalapackf/_results/.
For Windows, corresponding .zip examples are available including in %MKLROOT%/examples/examples_cluster_c.zip
There is a tension between the Visual Studio point release and Intel compilers on Windows.
The issue arises from Visual Studio frequently updating, and Intel compiler falling behind with the Intel compiler to Visual Studio ABI.
This can cause strange and unpredictable errors when compiling a C program, for example
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\include\vcruntime_string.h(18): error: expected an attribute name
_NODISCARD _Check_return_
Workaround
Intel support
notes
that a workaround is to force C++ compiler to be used with the /Tp<source_filename>option.
Meson
In Meson, this workaround can be applied in meson.build like:
project('foo', 'c')
cc = meson.get_compiler('c')
if cc.get_id() == 'intel-cl'
add_project_arguments('/TP', language: 'c')
message('applying workaround for Intel ICL bug with MSVC: https://www.scivision.dev/intel-compiler-windows-bug-workaround')
endif
CMake
In CMake, this workaround can be applied in CMakeLists.txt like:
cmake_minimum_required(VERSION3.0)project(fooC)if(WIN32ANDCMAKE_C_COMPILER_IDSTREQUALIntel) add_compile_options(/TP) message(STATUS"applying workaround for Intel ICL bug with MSVC: https://www.scivision.dev/intel-compiler-windows-bug-workaround")endif()
Minimal working examples of .appveyor.yml for both Windows and Linux using Python, C++ and/or Fortran are given below.
Use per-line distinct commands for each OS where needed.
The other way is to use matrix:, but unique per-line OS commands are more intuitive for many use cases.
For Linux, the stack: stanza picks software/version for a
set of common programs–here, Python 3.
Note the use of cmd: for Windows-exclusive commands, and sh: for Linux-exclusive commands.
As other CIs have done, AppVeyor
updated
to Python 3.8 in the default Linux,
MacOS and Windows images.
Since Python library wheels are often not immediately available for recently-released Python versions, this can lead to CI test nuisance errors at setup.
Many projects will experience AppVeyor
errors on Numpy setup
as AppVeyor attempts to compile Numpy since the wheel doesn’t exist yet for pre-release Python versions.
Select Python version
AppVeyor has several Python versions installed.
The AppVeyor Python version selection method is OS-dependent.
The examples are shown specific to Windows and Linux for didactic clarity.
They would normally be merged to a single .appveyor.yml.
Even minor point releases of Visual Studio can cause significant behavior changes and ABI breakages.
The full range of defaults that changes inside Visual Studio for a major release may be more than can be accommodated with user options.
Switching the Visual Studio “platform toolset” version may be of help.
Switch Visual Studio platform toolset version
In Visual Studio, click Project → Properties → Platform Toolset.
If the desired toolset is not present, use Visual Studio Installer to obtain it.
/permissive- VS2019 defaults
The standards-enforcing /permissive- flag is default in VS2019, but just turning that flag off may not be enough to compile projects from older MSVC versions.
The /permissive- flag is under Project → Properties → C/C++ →; Language → conformance mode.
Offline (non-Internet-connected) systems may experience failures on installing Python packages.
The package authors may not have CI setup for an offline test case, so they don’t realize it’s an issue.
In general, Python packages can use setup.py, setup.cfg and pyproject.toml more effectively in a Python-standard way to overcome these issues.
Install package before setup.py
Instead of telling users to manually install a package such as Numpy that’s required before fully running setup.py,
use pyproject.toml instead of setuptools setup_requires.
This is because currently, setuptools assumes the computer will be internet-connected and even if the package is already installed,
setup.py will fail on an offline system.
good example (most reliable and standard)
To ensure a package like Numpy is installed before setup.py fully runs, for example where f2py is used,
have a pyproject.toml file including:
This will auto-install the prereqs before setup.py runs.
When including this pyproject.toml parameter, do not omit “setuptools”, “wheel” or the package may fail to install.
PEP517 assumes unless told otherwise:
[build-system]
requires = ["setuptools", "wheel"]
such as by omitting them in a pyproject.toml file.
bad example (don’t do this)
These may fail on non-internet-connected systems, even if Numpy is already installed:
Pip is a widely-used, complex Python package installer program with a lot of legacy baggage.
The Python Software Foundation recognizes the critical need to update Pip, putting $116K to
sponsor a senior dev
to modernize Pip.
Sometimes, pip install fails to realize a .whl binary wheel is available.
Thus pip tries to download and install a package from source code.
In the case of a large package like SciPy, OpenCV or Pillow on an embedded system like the Raspberry Pi Zero,
it could take hours or even days to compile, probably failing numerous times due to missing prerequisite binary libraries.
Manual install wheel
A workaround to Pip not automatically finding the desired
.whl binary wheel
is to download it manually and install the .whl directly.
The binary wheels are often available at PyPi from the package download page, for example
SciPy.
For embedded systems such as Raspberry Pi, there may be non-PyPi sites such as
PiWheels.
Download the file, then pip install from the file like:
If the wheel binary is not compatible with the system, it will fail to import or run.
In that case, simply pip uninstall my_package_name and try something else.
Downloading binary Homebrew bottles without installing Homebrew can be useful to check the location of the bottle contents.
This is useful when developing Meson native-files or CMake Find*.cmake modules.
Since 2015, Homebrew distributes bottles from Bintray.
For example, HDF5 bottle may be inspected by:
tar --list -f hdf5-1.10.5_1.catalina.bottle.tar.gz
No Homebrew install is necessary for inspection, but actually using the libraries and binaries is of course best done by
installing Homebrew.
Usually it’s desired to check Python type annotations for an entire project.
However, when introducing type checking to a large Python project, it can be beneficial to introduce type checking gradually, to avoid massive refactoring all at once.
You will almost certainly find old code bugs due to type checking that will require refactoring, which introduces risk.
Mitigate this risk by introducing type checking gradually.
MyPy single file options
MyPy is a strongly recommended type checker, though others exist.
We typically use a per-project mypy.ini to make appropriate MyPy default behavior.
To tell MyPy to only check certain files, use the MyPy
–follow-imports=
option like: