How to use PyPi to distribute Python packages

Related: Replace/augment setup.py with pyproject.toml

This procedure is for PyPi Warehouse and requires secure, powerful twine uploader.

PyPi upload

This assumes you already have the setup.py and setup.cfg structure below.

Prereqs:

pip >= 10
setuptools >= 38.6
twine >= 1.11
wheel >= 0.31

one-time setup

Sign up for a PyPi account.

Install Twine

python -m pip install twine

Create ~/.pypirc with the content

[pypi]
username=pypiusername

Note that I did NOT save my password for security.

Upload

In your Python package directory, create thee source package and upload

python setup.py sdist

twine upload dist/*

Now the package is live to the world on PyPi. Anyone can install it via

python -m pip install myprogram

Minimal setup.py, setup.cfg

Assuming your package is named myprogram, have a directory structure like

setup.cfg
setup.py
myprogram/
   __init__.py

Minimal setup.py for PyPi:

from setuptools import setup; setup()

NOTE: you must always have a setuptools import when using setup.py to build distribution, EVEN IF you don’t explicitly use it. Use # noqa: F401 to quiet linters.

minimal setup.cfg for PyPi enabling Markdown README.

Pick from the list of classifiers suitable for your project.

You MUST increment the version number for each release, or PyPi will prevent upload

Error workarounds

error: Upload failed (400): Binary wheel ‘*-cp36-cp36m-linux_x86_64.whl’ has an unsupported platform tag ‘linux_x86_64’.

This error happens because currently only manylinux1 wheels are accepted by PyPi. It is not trivial to first start building manylinux1 wheels, so consider workaround: don’t upload the wheel, just the source code and your users will compile it on their machine.

python setup.py sdist
twine upload dist/*

Minimal Python setup.py with prerequisites

Related: Upload user Python module to PyPi

The commonly used setup.py for Python packages can be reduced to a one-line file for simple Python packages, by putting the project metadata in setup.cfg. The example setup.cfg file below is associated with a setup.py file containing merely:

from setuptools import setup; setup()

This is installed as usual by:

python -m pip install -e .

or similar.

It can be most effective to put all project configuration, including Python package prerequisites in setup.cfg instead of setup.py. setup.cfg is human-readable and machine-parseable without first installing the package. Putting as many parameters as possible into setup.cfg instead of setup.py is important and beneficial for reasons including:

  • reproducible results
  • security risk mitigation
  • getting package prerequisite tree list

Example

This is an example of best practices (since 2016) of minimal setup.py using setup.cfg. It does not use requirements.txt.

setup.cfg holds the machine-readable configuration, easy for humans too:

[metadata]
name = mycoolpkg
author = Joe Smith
author_email = me@gmail.com
description = My awesome program prints cool messages
version = file: __init__.py
url = https://github.com/joe/mycoolpkg
keywords =
  cool printing
  networking
classifiers =
  Development Status :: 4 - Beta
  Intended Audience :: Science/Research
  Programming Language :: Python :: 3.6
  Programming Language :: Python :: 3.7
  Programming Language :: Python :: 3.8
  Topic :: Scientific/Engineering
long_description = file: README.md
long_description_content_type = text/markdown
license_files =
  LICENSE.txt

[options]
python_requires = >= 2.7
setup_requires =
  setuptools >= 40.6
  pip >= 10
  wheel >= 0.31
packages = find:
zip_safe: False
scripts =
#  joesprint.py
install_requires =
#  colorama

[options.extras_require]
tests =
  pytest
  pytest-cov
  coveralls
  flake8
  mypy

[options.entry_points]
console_scripts =
#  joesprint = joesprint:main

Coverage

Test coverage is configured in .coveragerc, like:

[run]
cover_pylib = false
omit =
    /home/travis/virtualenv/*
    */site-packages/*

[report]
exclude_lines =
    pragma: no cover
    def __repr__
    RuntimeError
    NotImplementedError
    FileNotFoundError
    ImportError

PEP8 / Type hinting checks

PEP8 checking via flake8 is configured in .flake8:

[flake8]
max-line-length = 132
exclude = .git,__pycache__,doc/,docs/,build/,dist/,archive/

MyPy type hint checking is configured via mypy.ini.

pytest.ini

If one wishes to configure Pytest itself, that can be done in pytest.ini. It is recommended to configure Pytest in pytest.ini instead of setup.cfg for best stability. For example, require Pytest ≥ 3.9 with pytest.ini containing:

[pytest]

minversion = 3.9

MANIFEST.in

MANIFEST.in is usually not necessary. It’s better in general to specify scripts and data files in setup.cfg.

Notes

  • install_requires cannot read requirements.txt as file: is not in the setup.cfg install_requires input types
  • console_scripts expects a file joesprint.py with the function main designed to accept command line input, perhaps using argparse.ArgumentParser
  • python_requires parameter avoids user confusion if they have an old Python version. Instead of them opening a GitHub issue or just not using your program at all, they’ll get a useful error message.

Classifiers

Classifiers are optional, but help your project be indexed better in PyPi (and hence search engines). Classifiers must be from this official classifiers list or they will fail when uploading your package to PyPi.

setup.py Fortran f2py

Python can easily import Fortran code using f2py. See this f2py example setup.py.

Notes

setuptools 38.6 and wheel 0.31 support Markdown README. Pip 10 brought pyproject.toml support, necessary for clean handling of Python Extension Modules via Numpy as well as setup.cfg support.

Install Xrdp for VNC via Windows Remote Desktop

xrdp creates an RDP server on remote Linux PCs.

Setup RDP client

RDP client (on your laptop) is installed by:

  • Windows: factory installed
  • Mac: RDP client
  • Linux: apt install xfreerdp

Setup Xrdp server

The remote Linux PC has the Xrdp server. Install Xrdp and Openbox desktop

apt install xrdp openbox

Create ~/.xsession containing

exec openbox-session

Enable xrdp with new config

service xrdp restart

Notes

  • Openbox will show a grey screen upon typing password at Xrdp login. Right-click mouse to open menu.
  • If you just get a gray/black screen, try editing /etc/xrdp/startwm.sh to contain
#!/bin/sh

if [ -r /etc/default/locale ]; then
. /etc/default/locale
export LANG LANGUAGE
fi

exec openbox-session

MyPy Python static type hinting quick start

The benefits of Python static type checking and examples have already been given at length. In summary, Python static type checking enhances code quality now and in the future by defining (constraining) variables and functions (methods). Type hinting is compatible with Python 2 when the correct syntax is used, and MyPy can check type hinting in Python 2 mode.

assert vs. type hinting

Type enforcement can be done with assert, but type hinting is much more concise, flexible and readable than assert. With type hinting, the hint is right at the variable name (e.g. in the function declaration), while assert must occur in the code body.

Duck typing

MyPy static type checker considers the following to be interchangeable (valid) due to duck typing:

  • int ↔ float
  • float ↔ complex

Note that str is not equivalent to bytes, one of the major benefits of Python 3.

Install

pip install mypy

Usage

From the code project top level:

mypy .

configure mypy.ini to eliminate nuisance errors or otherwise configure mypy.

It takes a little practice to understand the messages. Where multiple types are accepted, for example, str and pathlib.Path use typing.Union. See the examples below.

Examples

Many times a function argument can handle more than one type. This is handled as follows:

from typing import Union
from pathlib import Path


def reader(fn: Union[Path, str]) -> str:
    fn = Path(fn).expanduser()

    txt = fn.read_text()

    return txt

Another case is where lists or tuples are used, the types within can be checked (optionally):

from typing import Union, Tuple, List


def reader(fn: Union[Path, str]) -> Tuple[float, float]:
    fn = Path(fn).expanduser()

    txt: List[str] = fn.read_text().split(',')

    latlon = (float(txt[0]), float(txt[1]))

    return latlon

Or perhaps dictionaries, where optionally types within can be checked:

from typing import Union, Dict, List, Any


def reader(fn: Union[Path, str]) -> Dict[str, float]:
    fn = Path(fn).expanduser()

    txt: List[str] = fn.read_text().split(',')

    params = {'lat': float(txt[0]),
              'lon': float(txt[1])}

    return params

If many value types are in the dictionary, you can use Union[] or simply Any e.g.

Dict[str, Any]

The default where no type is declared is Any, which basically means “don’t check this variable at this location in the code”.

Find the path of an executable in Python

A good way to use executables on the system Path, auto-adding the .exe extension on Windows is by Python shutil.which

import shutil
import subprocess

# None if executable not found
exe = shutil.which('ls')

cmd = [exe, '-l']
print(cmd)

subprocess.run(cmd)

The fact that shutil.which() returns None for non-found executables makes it convenient for use with Pytest

Specify path

For non-system utilities or other programs not on PATH, where the executable path is known do like:

shutil.which('myexe', path=str(mypath))

Fixed in Python 3.8, but present in earlier Python versions is a bug that requires a string for shutil.which(..., path=str(mypath))–pathlib.Path won’t work until Python 3.8 for that instance.

Test python setup.py install using Travis-CI

For continuous integration, it’s important to test

python setup.py install

along with the more commonly used in situ development mode

pip install -e .

Here’s an example .travis.yml installing from extras_requires while using python setup.py install:

language: python
group: travis_latest
dist: xenial

git:
  depth: 25
  quiet: true

python:
- 3.7
- 3.6

matrix:
  include:
  - os: linux
    name: Install integration
    python: 3.7
    install:
    - python setup.py install
    - pip install $(basename $TRAVIS_REPO_SLUG)[tests]
    script:
    - cd $HOME
    - python -m pytest $TRAVIS_BUILD_DIR/tests


install: pip install -e .[tests]

script: pytest -r a -v

The extra steps in the “install integration” are to ensure the install under site-packages is used instead of the local directory.

To be clear for those not yet familiar with Travis-CI build matrix, this .travis.yml will run 3 tests:

  • Python 3.6 development mode
  • Python 3.7 development mode
  • Python 3.7 integration via installed package

$(basename $TRAVIS_REPO_SLUG) assumes your repo name is the same as the Python package name. It uses Travis-CI default environment variables Set this manually to the Python package name if different than the repo name.

nmap in Cygwin - seamlessly

The original Windows Subsystem for Linux (2016 - 2019) doesn’t work with Nmap. At the time of this writing, it’s not yet known if the “new” WSL2 will work with Nmap or not. On Windows, use Nmap from Windows itself or via Cygwin (which is just calling native Windows Nmap).

Install

  1. Download nmap
  2. Install nmap “self-installer” .exe. When asked, also install Npcap.
  3. Cygwin: add to ~/.bash_profile the following. Note the \ and \( are vital for Cygwin shell to interpret the command correctly.
alias nmap="/cygdrive/c/Program\ Files\ \(x86\)/Nmap/nmap.exe"

Open a new Cygwin window to start using nmap

Test

nmap 8.8.8.8

results in

Starting Nmap ( https://nmap.org ) 
Nmap scan report for 8.8.8.8
Host is up (0.0092s latency).
Not shown: 998 filtered ports
PORT STATE SERVICE
53/tcp open domain
443/tcp open https

Nmap done: 1 IP address (1 host up) scanned in 7.41 seconds

Troubleshooting

  • errors about interface → try running Cygwin as Administrator (right click on Cygwin icon).
  • find interface names available to nmap
nmap --iflist

Notes

  • to find servers with a particular port open on a subnet, try my Python findssh program that scans for servers without nmap.
  • If you don’t install Npcap when asked in the nmap installer, nmap does not work. Nmap claimed no host existed at a known working IP address.

Why isn’t nmap built into Cygwin?

nmap requires deeper access to the Windows networking stack that would go beyond the normal scope of Cygwin. Note that Nmap itself needs the separate program Npcap (forked from WinPcap) to work on Windows.

Setup GitHub Keybase.io PGP signed/verified commit

With recent cybersecurity scandals over user modules written in Python and other languages, it’s past time to employ (and even require, via per-repo GitHub Branch Rules) signed / verified commits at GitHub. PGP IDs can be readily tied between GitHub, online personality at Twitter, website, etc. via the free Keybase.io service.

Setup

This process assumes:

0. GPG install

  • Linux: apt install gnupg
  • MacOS: brew install gnupg

On Windows you can setup GPG via “Git Bash” just like Linux (easy). Or on Windows you can get GPG via Kleopatra GPG binary install.

1. Export Keybase public & private key and import into GPG:

Linux / MacOS / Windows Subsystem for Linux / Git Bash:

keybase pgp export | gpg --import

keybase pgp export --secret | gpg --allow-secret-key --import

Windows Kleopatra:

keybase pgp export > keybase-public.asc

keybase pgp export --secret > keybase-private.asc

The “keybase-private.asc” will be itself encrypted via password you enter–must be distinct from your Keybase password.

With Kleopatra, import keybase-private.asc

2. Verify key

Linux / MacOS / Windows Subsystem for Linux / Git Bash:

gpg --list-secret-keys --keyid-format LONG

one of the first lines will be like:

sec   rsa4096/05F2BD2A525007DF

copy the hexadecimal part after the /. This is a public reference to keybase.io keypair. It’s shown on the keybase.io public profile, next to the key icon.

Windows Kleopatra:

In Kleopatra, right click the key in the list to “certify” the key. Note that the rightmost part of the fingerprint matchs the public reference to keybase.io keypair. It’s shown on the keybase.io public profile, next to the key icon.

3. Add GitHub verified email

At least one of these GitHub verified email address MUST match the [user] email in ~/.gitconfig or Unverified warnings appear on GitHub commits!

Linux / MacOS / Windows Subsystem for Linux / Git Bash:

For this example I use my GPG public ID–you use yours.

gpg --edit-key 05F2BD2A525007DF

In the interactive GPG session that launches, type

adduid

and enter Name and the Email address–which must exactly match the GitHub verified email address. I also add the @users.noreply.github.com fake email that I always use to avoid spam. Do adduid twice–once for the real GitHub verified email address and again for the github_username@users.noreply.github.com fake email.

Add “trust” from the GPG> prompt:

trust

Since it’s you, perhaps a trust level of 5 is appropriate. type

save

to save changes, which may not show up until exiting and reentering the GPG> prompt.

Windows Kleopatra:

In Kleopatra, right click the key and add email addresses via “Add User ID”. Do this twice–once for the real GitHub verified email address and again for the github_username@users.noreply.github.com fake email.

4. Configure Git to use Keybase

From Terminal / Command Prompt:

Do this using your public Keybase hex ID as seen next to the key logo on your public Keybase.io profile, not mine in the example below.

git config --global user.signingkey 05F2BD2A525007DF

git config --global commit.gpgsign true
  • Windows Kleopatra: additionally, point Git to GPG: git config --global gpg.program "C:\Program Files (x86)\GnuPG\bin\gpg.exe"

check ~/.gitconfig to see entries under [user] signingkey and [commit] gpgsign

Add the GPG public key to GitHub–copy and paste the output from this command into the GitHub New GPG Key

Linux / MacOS / Windows Subsystem for Linux / Git Bash:

gpg --armor --export 05F2BD2A525007DF

Windows Kleopatra:

Export public certificate to file and copy/paste to GitHub New GPG Key

Verify

Make a git commit after the procedure above, and see the signature notes:

git log --show-signature

it will start with

gpg: Signature made

Temporary disable signing

If you temporarily lose access to your GPG password, you won’t be able to git commit. A temporary workaround is to edit ~/.gitconfig to have

[commit]
    gpgsign = false

or simply add the --no-gpg-sign option like:

git commit -am "msg" --no-gpg-sign

Alternatively, if you prefer not signing as default, you can sign only certain commits by

git commit -S

Note that’s a capital S.

Notes

reference 1