xarray.open_dataset
gained the ability to
open HDF5 files in 0.12.0.
However, this can cause Python to quietly crash without error message, which can be quite confusing.
This is true even with the minimum required versions of xarray, h5py and h5netcdf installed.
We don’t have a specific workaround for this other than to use
h5py
directly.
We have used h5py for several years in high-stakes operations, including data analysis and data collection.
Both h5py and netcdf4 Python modules work with
context managers
to avoid excess I/O resource consumption.
Since GitLab Community Edition is
open source,
large projects like CMake may host their own self-managed GitLab instance.
To make merge requests to such projects, one can configure Git SSH.
For this example, we use Kitware’s CMake GitLab instance https://gitlab.kitware.com/cmake.
This procedure works for any operating system, including Windows.
First, create an account on the self-managed GitLab instance and fork the desired repo.
This will be available like
One should almost always use a Python
context manager
when working with file I/O in Python.
Context managers for Python I/O resources help avoid exceeding system
resource limits.
For long running jobs, context managers help avoid random crashes due to excess file I/O resource utilization from files left hanging open.
There are edge cases where you do need to keep the handle open without context manager–for example, inside a for loop.
In many cases, it may be better and easier to let the file open and close with the context manager.
It is also possible to create your own content managers with Python
contextlib,
which we use in
georinex
for example.
Context Manager examples
These examples assume you’ve done something like
from pathlib import Path
fn = Path('~/mydir/myfile').expanduser()
SSH Agent remembers SSH Public Key authentication for a period of time.
While native Windows has SSH built in, there is no straightforward way to have an SSH agent in Windows itself.
Instead, one can use WSL for SSH agent as follows.
SSH agent setup
This works for Linux in general, including Windows Subsystem for Linux.
Python on Windows can be used with Windows Subsystem for Linux.
Using Python on WSL can be advantageous because of easier compiler access.
This procedure includes the graphical Spyder IDE if you so desire.
Install Python in WSL
These commands are all from the Linux / WSL Terminal.
The Free TightVNC server works with Ubuntu and other modern Linux distros.
In general the default 3-D Ubuntu desktop is not available over VNC, so we use a traditional 2-D desktop environment over VNC.
Setup VNC server
Install Linux VNC Server:
apt install tightvncserver openbox
Choose desktop environment on server PC–XFCE4, Openbox or other you prefer.
Pick ONE of the following:
Openbox is the lightest-weight desktop environment–right click to open a menu.
Otherwise, you just have a plain gray background, extremely minimal–good for embedded systems and old PCs.
Openbox-Message: Unable to find a valid menu file “/var/lib/openbox/debian-menu.xml”
Wwhen I get this error, I’m also unable to open a terminal.
You can leave your VNC desktop running – it is not the same as your local desktop.
It is a little tricky to share your local desktop reliably–I have done so with X11VNC, but it can be more trouble than it is worth!
It’s MUCH easier to startup a new separate desktop session with vncserver or x11vnc
After installing a new desktop environment, at your next local login, you’ll need to rechoose the traditional Ubuntu desktop (it will then remember this choice).
Meson does not have built-in the ability to download any file.
While this could also be done via a custom_target(), we do it via run_command() in meson.build.
This technique uses only Python stdlib modules; no extra pip install is needed.
#!/usr/bin/env python3"""
We use SystemExit as this will not blast the whole traceback to Meson.
Usually just a terse stderr will suffice and not overwhelm the Meson user.
"""from pathlib import Path
import urllib.request
import urllib.error
import hashlib
import argparse
import typing
import socket
defurl_retrieve(
url: str,
outfile: Path,
filehash: typing.Sequence[str] = None,
overwrite: bool = False,
):
"""
Parameters
----------
url: str
URL to download from
outfile: pathlib.Path
output filepath (including name)
filehash: tuple of str, str
hash type (md5, sha1, etc.) and hash
overwrite: bool
overwrite if file exists
"""
outfile = Path(outfile).expanduser().resolve()
if outfile.is_dir():
raiseValueError("Please specify full filepath, including filename")
# need .resolve() in case intermediate relative dir doesn't existif overwrite ornot outfile.is_file():
outfile.parent.mkdir(parents=True, exist_ok=True)
try:
urllib.request.urlretrieve(url, str(outfile))
except (socket.gaierror, urllib.error.URLError) as err:
raiseSystemExit(
"ConnectionError: could not download {} due to {}".format(url, err)
)
if filehash:
ifnot file_checksum(outfile, filehash[0], filehash[1]):
raiseSystemExit("HashError: {}".format(outfile))
deffile_checksum(fn: Path, mode: str, filehash: str) -> bool:
h = hashlib.new(mode)
h.update(fn.read_bytes())
return h.hexdigest() == filehash
if __name__ =="__main__":
p = argparse.ArgumentParser()
p.add_argument("url", help="URL to file download")
p.add_argument("outfile", help="filename to download to")
p.add_argument("-hash", help="expected hash", nargs=2)
P = p.parse_args()
url_retrieve(P.url, P.outfile, P.hash)
def extract_zip(fn: Path, outpath: Path, overwrite: bool = False):
outpath = Path(outpath).expanduser().resolve()
# need .resolve() in case intermediate relative dir doesn’t exist
if outpath.is_dir() and not overwrite:
return
fn = Path(fn).expanduser().resolve()
with zipfile.ZipFile(fn) as z:
z.extractall(str(outpath.parent))
def extract_tar(fn: Path, outpath: Path, overwrite: bool = False):
outpath = Path(outpath).expanduser().resolve()
# need .resolve() in case intermediate relative dir doesn’t exist
if outpath.is_dir() and not overwrite:
return
fn = Path(fn).expanduser().resolve()
if not fn.is_file():
raise FileNotFoundError(fn) # keep this, tarfile gives confusing error
with tarfile.open(fn) as z:
z.extractall(str(outpath.parent))
if name == “main“:
p = argparse.ArgumentParser()
p.add_argument(“infile”, help=“compressed file to extract”)
p.add_argument(“outpath”, help=“path to extract into”)
P = p.parse_args()
infile = Path(P.infile)
if infile.suffix.lower() == ".zip":
extract_zip(infile, P.outpath)
elif infile.suffix.lower() in (".tar", ".gz", ".bz2", ".xz"):
extract_tar(infile, P.outpath)
else:
raise ValueError("Not sure how to decompress {}".format(infile))
Python’s
urllib.request.urlretrieve
doesn’t have a way to handle connection timeouts.
This can lead to user complaints where they think your program is hanging, when really it’s a bad internet connection since urlretrieve will hang for many minutes.
Python requests download files
This is a robust way to download files in Python with timeout.
I name it url_retrieve to remind myself not to use the old one.
from pathlib import Path
import requests
defurl_retrieve(url: str, outfile: Path):
R = requests.get(url, allow_redirects=True)
if R.status_code !=200:
raise ConnectionError('could not download {}\nerror code: {}'.format(url, R.status_code))
outfile.write_bytes(R.content)
Why isn’t this in requests?
Because the Requests BDFL
doesn’t want it
pure Python download files
If you can’t or don’t want to use requests, here is how to download files in Python using only built-in modules:
from pathlib import Path
import typing
import urllib.request
import urllib.error
import socket
defurl_retrieve(
url: str,
outfile: Path,
overwrite: bool = False,
):
"""
Parameters
----------
url: str
URL to download from
outfile: pathlib.Path
output filepath (including name)
overwrite: bool
overwrite if file exists
"""
outfile = Path(outfile).expanduser().resolve()
if outfile.is_dir():
raiseValueError("Please specify full filepath, including filename")
# need .resolve() in case intermediate relative dir doesn't existif overwrite ornot outfile.is_file():
outfile.parent.mkdir(parents=True, exist_ok=True)
try:
urllib.request.urlretrieve(url, str(outfile))
except (socket.gaierror, urllib.error.URLError) as err:
raise ConnectionError(
"could not download {} due to {}".format(url, err)
)