nmap in Cygwin - seamlessly

The original Windows Subsystem for Linux (2016 - 2019) doesn’t work with Nmap. At the time of this writing, it’s not yet known if the “new” WSL2 will work with Nmap or not. On Windows, use Nmap from Windows itself or via Cygwin (which is just calling native Windows Nmap).

Install

  1. Download nmap
  2. Install nmap “self-installer” .exe. When asked, also install Npcap.
  3. Cygwin: add to ~/.bash_profile the following. Note the \ and \( are vital for Cygwin shell to interpret the command correctly.
alias nmap="/cygdrive/c/Program\ Files\ \(x86\)/Nmap/nmap.exe"

Open a new Cygwin window to start using nmap

Test

nmap 8.8.8.8

results in

Starting Nmap ( https://nmap.org ) 
Nmap scan report for 8.8.8.8
Host is up (0.0092s latency).
Not shown: 998 filtered ports
PORT STATE SERVICE
53/tcp open domain
443/tcp open https

Nmap done: 1 IP address (1 host up) scanned in 7.41 seconds

Troubleshooting

  • errors about interface → try running Cygwin as Administrator (right click on Cygwin icon).
  • find interface names available to nmap
nmap --iflist

Notes

  • to find servers with a particular port open on a subnet, try my Python findssh program that scans for servers without nmap.
  • If you don’t install Npcap when asked in the nmap installer, nmap does not work. Nmap claimed no host existed at a known working IP address.

Why isn’t nmap built into Cygwin?

nmap requires deeper access to the Windows networking stack that would go beyond the normal scope of Cygwin. Note that Nmap itself needs the separate program Npcap (forked from WinPcap) to work on Windows.

Setup GitHub Keybase.io PGP signed/verified commit

With recent cybersecurity scandals over user modules written in Python and other languages, it’s past time to employ (and even require, via per-repo GitHub Branch Rules) signed / verified commits at GitHub. PGP IDs can be readily tied between GitHub, online personality at Twitter, website, etc. via the free Keybase.io service.

Setup

This process assumes:

0. GPG install

  • Linux: apt install gnupg
  • MacOS: brew install gnupg

On Windows you can setup GPG via “Git Bash” just like Linux (easy). Or on Windows you can get GPG via Kleopatra GPG binary install.

1. Export Keybase public & private key and import into GPG:

Linux / MacOS / Windows Subsystem for Linux / Git Bash:

keybase pgp export | gpg --import

keybase pgp export --secret | gpg --allow-secret-key --import

Windows Kleopatra:

keybase pgp export > keybase-public.asc

keybase pgp export --secret > keybase-private.asc

The “keybase-private.asc” will be itself encrypted via password you enter–must be distinct from your Keybase password.

With Kleopatra, import keybase-private.asc

2. Verify key

Linux / MacOS / Windows Subsystem for Linux / Git Bash:

gpg --list-secret-keys --keyid-format LONG

one of the first lines will be like:

sec   rsa4096/05F2BD2A525007DF

copy the hexadecimal part after the /. This is a public reference to keybase.io keypair. It’s shown on the keybase.io public profile, next to the key icon.

Windows Kleopatra:

In Kleopatra, right click the key in the list to “certify” the key. Note that the rightmost part of the fingerprint matchs the public reference to keybase.io keypair. It’s shown on the keybase.io public profile, next to the key icon.

3. Add GitHub verified email

At least one of these GitHub verified email address MUST match the [user] email in ~/.gitconfig or Unverified warnings appear on GitHub commits!

Linux / MacOS / Windows Subsystem for Linux / Git Bash:

For this example I use my GPG public ID–you use yours.

gpg --edit-key 05F2BD2A525007DF

In the interactive GPG session that launches, type

adduid

and enter Name and the Email address–which must exactly match the GitHub verified email address. I also add the @users.noreply.github.com fake email that I always use to avoid spam. Do adduid twice–once for the real GitHub verified email address and again for the github_username@users.noreply.github.com fake email.

Add “trust” from the GPG> prompt:

trust

Since it’s you, perhaps a trust level of 5 is appropriate. type

save

to save changes, which may not show up until exiting and reentering the GPG> prompt.

Windows Kleopatra:

In Kleopatra, right click the key and add email addresses via “Add User ID”. Do this twice–once for the real GitHub verified email address and again for the github_username@users.noreply.github.com fake email.

4. Configure Git to use Keybase

From Terminal / Command Prompt:

Do this using your public Keybase hex ID as seen next to the key logo on your public Keybase.io profile, not mine in the example below.

git config --global user.signingkey 05F2BD2A525007DF

git config --global commit.gpgsign true
  • Windows Kleopatra: additionally, point Git to GPG: git config --global gpg.program "C:\Program Files (x86)\GnuPG\bin\gpg.exe"

check ~/.gitconfig to see entries under [user] signingkey and [commit] gpgsign

Add the GPG public key to GitHub–copy and paste the output from this command into the GitHub New GPG Key

Linux / MacOS / Windows Subsystem for Linux / Git Bash:

gpg --armor --export 05F2BD2A525007DF

Windows Kleopatra:

Export public certificate to file and copy/paste to GitHub New GPG Key

Verify

Make a git commit after the procedure above, and see the signature notes:

git log --show-signature

it will start with

gpg: Signature made

Temporary disable signing

If you temporarily lose access to your GPG password, you won’t be able to git commit. A temporary workaround is to edit ~/.gitconfig to have

[commit]
    gpgsign = false

or simply add the --no-gpg-sign option like:

git commit -am "msg" --no-gpg-sign

Alternatively, if you prefer not signing as default, you can sign only certain commits by

git commit -S

Note that’s a capital S.

Notes

reference 1

AppVeyor Python 3.8 temporarily broken

AppVeyor updated to pre-release Python 3.8 in their default Ubuntu1804 image. Since Python library wheels are often not available for beta Python versions, this can lead to CI test nuisance errors at setup. Many projects will experience AppVeyor errors on Numpy setup as AppVeyor attempts to compile Numpy since the wheel doesn’t exist yet for pre-release Python versions.

Select Python version

AppVeyor has several Python versions installed. The AppVeyor Python version selection method is OS-dependent.

The examples are shown specific to Windows and Linux for didactic clarity. They would normally be merged to a single .appveyor.yml.

Windows builds

Select Python version on AppVeyor Windows CI by setting .appveyor.yml to the desired AppVeyor Windows Python version directory:

image:
- Visual Studio 2017

environment:
  PY_DIR: C:\Python37-x64

init:
- cmd: set PATH=%PY_DIR%;%PY_DIR%\Scripts;%PATH%

Linux builds

Select Python version on AppVeyor Linux CI by setting .appveyor.yml to the desired AppVeyor Linux Python version directory:

image:
- Ubuntu1804

stack: python 3.7

AppVeyor for Windows and Linux builds

Related: f2py on AppVeyor Windows


AppVeyor Linux-based CI works like the free Windows-based CI AppVeyor is well known for. For MacOS Travis CI is a prime choice.

Minimal Examples

Minimal working examples of .appveyor.yml for both Windows and Linux using Python, C++ and/or Fortran are given below. Use per-line distinct commands for each OS where needed. The other way is to use matrix:, but unique per-line OS commands are more intuitive for many use cases.

  • For Linux, the stack: stanza picks software/version for a set of common programs–here, Python 3.
  • Note the use of cmd: for Windows-exclusive commands, and sh: for Linux-exclusive commands.

Python

Note: f2py on AppVeyor Windows has its own page–importing Fortran libraries from Python.

This appveyor.yml runs Python 3.7 on Windows 10 and Ubuntu 18.04:

image:
- Visual Studio 2017
- Ubuntu1804

stack: python 3.7

environment:
  PY_DIR: C:\Python37-x64

clone_depth: 25

build: off

init:
- cmd: set PATH=%PY_DIR%;%PY_DIR%\Scripts;%PATH%

install:
- python --version
- python -m pip install -e .[tests,full]

test_script: python -m pytest -r a -v

Fortran

image:
- Visual Studio 2017
- ubuntu1804

environment:
  MINGW_DIR: C:\mingw-w64\x86_64-7.2.0-posix-seh-rt_v5-rev1\mingw64\bin

clone_depth: 25

build: off

init:
- cmd: set PATH=%MINGW_DIR%;%PATH%

install:
- cd bin
- cmd: cmake -G "MinGW Makefiles" -DCMAKE_SH="CMAKE_SH-NOTFOUND" ..
- sh: cmake ..

- cmake --build . --target install

test_script:
- ctest -V

Manual Linux install

Install additional Linux programs in AppVeyor via commands like:

init:
- sh: sudo apt -yq update > /dev/null
- sh: sudo apt install -yq --no-install-suggests --no-install-recommends bwbasic > /dev/null

Notes


Command “python setup.py egg_info” failed with error code 1

Try using Miniconda3 C:\Miniconda3-x64

Fix Python 3 on Windows error Microsoft Visual C++ 14.0 is required

Related: Fix Python 2 error Visual C++ 10.0 missing vcvarsall.bat


Fix the error for Python on Windows:

error Microsoft Visual C++ 14.0 is required

as follows. Visual Studio Studio Build Tools requires about 3 GB of disk space.

Python versionVisual Studio
2.7, 3.42010
3.52015 / 2017
3.6, 3.7, 3.82017 / 2019

In mid-2019, most people will want VS2017, as that’s what most software is compatible with. We are seeing increasing adoption of VS2019 as we go into the second half of 2019.

  1. Install using any ONE of these choices:
  2. Select: Workloads → C++ build tools.
  3. Install options: select only the “Windows 10 SDK” (assuming the computer is Windows 10). Optionally, if you want to use MSVC cl.exe C/C++ compiler from the command line, additionally select the C++ build tools, which takes an additional 2 GB disk space.

Notes

Windows Python needs Visual C++ libraries installed via the SDK to build code, such as via setuptools.extension.Extension or numpy.distutils.core.Extension. For example, building f2py modules in Windows with Python requires Visual C++ SDK as installed above. On Linux and Mac, the C++ libraries are installed with the compiler.

Reference

This was the former link to Microsoft Visual C++ Build Tools–no longer active: https://landinghub.visualstudio.com/visual-cpp-build-tools

Specifying OpenCV as Python package prereq

Related: Install OpenCV in Python


An easy way to install OpenCV in Python is simply:

python -m pip install opencv-python

This carries over into your setup.cfg file, simply include:

[options]
setup_requires =
  setuptools >= 38.6
  pip >= 10
  wheel >= 0.31

[options.extras_require]
opencv =
  opencv-python

This allows users to retain their existing OpenCV install, which may be customized / optimized, while giving an easy path for new OpenCV users. You can tell users that if they don’t already have OpenCV, try:

pip install -e .[opencv]

CI

This is also a good technique for continuous integration systems like Travis-CI, in the .travis.yml file include:

install:
- pip install -e .[tests,cov,opencv]

just takes a few seconds.

Update Raspberry Pi to Raspbian Buster 10

Related: Full headless Raspberry Pi setup


Why upgrade

It’s worthwhile to upgrade to Debian Buster 10 to have Python 3.7 with system / GPIO libraries not always easy to install otherwise. It is generally not worthwhile to upgrade Raspbian in place–just copy off your files, do a fresh Debian install, and copy your files back. Keep the original SD card in case some setup is hard to figure out.

The best way to upgrade the Raspberry Pi operating system is get a new SD card and load with Raspbian from scratch as described below.

Copy any wanted files from Raspberry Pi ~ home directory to your PC with sftp or rsync. This is in case the original SD card fails.

The contents of the SD card are erased in the following procedure. We suggest getting a new SD card so that you have the original SD card with working configuration in case there’s an issue.

Download Raspbian Image ~ 400 Mbytes

wget https://downloads.raspberrypi.org/raspbian_lite_latest

Write Raspbian image to SD card. Before and after inserting the SD card into your PC, type

lsblk

Assuming the SD card was at /dev/mmcblk0 (use lsblk to find SD card mount point):

unzip -p *raspbian*lite* | sudo dd of=/dev/mmcblk0 bs=4M status=progress conv=fsync

The option conv=fsync prevents dd from instantly saying it’s done, helping avoid mistakenly removing the SD card before it’s done writing. This will take 5-10 minutes depending on your SD card speed.

sync

ensures the writing has completed.

Enable SSH by changing to the SD card boot directory, and type

touch ssh

before physically ejecting the SD card from your PC.

Headless Raspberry Pi first boot

Without a monitor and keyboard on the Pi, the Avahi dæmon on the Raspberry Pi will make it easy to find your Raspberry Pi when you plug into your LAN.

Note: changing the hostname of your Raspberry Pi also changes the Avahi address correspondingly.

Put the SD card into the Pi, plug an Ethernet cord into your LAN router/switch and from your PC:

ssh pi@raspberrypi.local

Update Pi software

apt install rpi-update
rpi-update
apt update
apt upgrade

Raspberry Pi Configuration

raspi-config

Raspberry Pi reliable long-term install tips

Avoid overclocking beyond one step above minimum. Use heatsink kit for high-stakes remote operations.

Check temperature of Raspberry Pi CPU

Related: Monitor Raspberry Pi DC input voltage


Check Raspberry Pi CPU Temperature:

/opt/vc/bin/vcgencmd measure_temp

A light-moderate use Raspberry Pi 2 in an office environment measures in the 40 - 45 ℃ range (no fan, small heatsink, with case).

A red thermometer icon GPU-superimposed on the Raspberry Pi display output means the Raspberry Pi is overheating and is throttling the CPU and GPU to avoid self-destruction.

CPU temperature [C]iconthrottle
< 80nonenone
80 - 85half full thermometer redCPU
> 85full thermometer redCPU & GPU

CPU clock

The current Raspberry Pi CPU speed (actual speed, not limits or requested speed) is obtained from

vcgencmd measure_clock arm

The output is in units of Hertz:

frequency(45)=1199998000

The Raspberry Pi CPU clock speed and power consumption is dynamic:

clock speed [MHz]Raspi 2Raspi 3Raspi 3+Raspi 4
idle600600600600
100% one or more cores900120014001500

Log CPU/temperature

crontab -e can periodically log temperature and CPU frequency, e.g. add a line like:

@hourly  vcgencmd measure_clock arm > /home/pi/temperature.log

Raspberry Pi vs. Beaglebone Black vs. Intel Edison

Pi ModelCPURAMfeatures
ZeroARMv6512 MB LPDDR2low power consumption, single-core CPU.
4ARMv8 (A72)1..4 GB LPDDR4quad-core CPU, GPU, Bluetooth 5, 802.11ac, gigabit Ethernet, USB 3.0

The benchmarks for Raspberry Pi 2 and Raspberry Pi 3 show far faster performance than the Raspberry Pi 1 / Zero. Using the graphical desktop, the Raspberry Pi 4 is useful for modest desktop use. The new CPU-direct USB and Ethernet of the Raspberry Pi 4 brings drastically faster IO.

Beaglebone

Beaglebone’s deterministic IO due to the PRU is a key advantage for certain applications. The Beaglebone DDR3L RAM, and ARMv7 CPU is nearly twice as powerful as the Raspberry Pi Zero, native Ethernet, onboard SSD (eMMC), better onboard I/O (for local sensors), etc. However, CPU or GPU demanding tasks should consider the latest Raspberry Pi or Intel NUC, etc. instead.

Intel Edison

Intel Edison was a better choice for energy-budget conscious tasks than the Raspberry Pi and the Beaglebone Black due to the Edison’s dual-core x86 CPU, fast RAM, and extremely favorable energy efficiency. The Edison may have run 10-20x as long or more on battery than most Raspberry Pi models or Beaglebone Black.

The Raspberry Pi Zero has a fairly limited ARMv6 single core CPU, but much lower idle power for battery-powered applications.

Former Raspberry Pi models

These models shouldn’t generally be used anymore unless there’s a specific reason:

Pi ModelCPURAMusefulness
1ARMv6512 MB LPDDR2high power consumption, single-core CPU.
2ARMv8 (A53)1 GB LPDDR2no onboard wireless.
3+ARMv8 (A53)1 GB LPDDR2quad-core CPU, GPU, Bluetooth 4.2 / 802.11ac, gigabit Ethernet

Compile/install Python 3.8 on Raspberry Pi

Raspbian Buster 10 for Raspberry Pi includes Python 3.7. Here is how to compile Python 3.8 on the Raspberry Pi.

If you need to use apt installed Python modules that access hardware like GPIO, you can always access system Python 3.7 via /usr/bin/python3

Compile Python interpreter

In general, one of the few times you should ever use sudo on a Linux computer is apt install.

Get prereqs on your ARM device:

apt install libffi-dev libbz2-dev liblzma-dev libsqlite3-dev libncurses5-dev libgdbm-dev zlib1g-dev libreadline-dev libssl-dev tk-dev build-essential libncursesw5-dev libc6-dev openssl git

Extract the latest Python source code

Configure (3 minutes on Raspberry Pi 2):

cd cpython-3.8*

./configure --prefix=$HOME/.local --enable-optimizations

Build and install–this step takes 10-40 minutes, depending on Raspberry Pi model. Do not use sudo!

make -j -l 4
make install

Note: don’t omit -l 4 or Pi will be quickly overwhelmed and error build. This limits load average to 4. Without it, load average will soar to 100+ (bad).

Add to ~/.bashrc:

export PATH=$HOME/.local/bin/:$PATH

then open a new Terminal

Verify

Check that which python3 and which pip3 etc. refer to ~/.local/bin/ instead of /usr/bin. Don’t uninstall system Python 3 /usr/bin/python3 because system packages depend on it. The PATH you set in Step 5 above makes Linux prefer the new Python.

Notes

reference

Why upgrade Python?

Python 3.8 brings further speed / efficiency improvements. Specific benefits include: