Fix BibTeX error with .bbl file

Sometimes cryptic errors occur if there was a syntax error in a .bib BibTeX bibliography file that doesn’t disappear even when the .bib syntax is corrected. The fix for this is often to delete the auto-generated files:

Example

Suppose the top-level LaTeX file is “main.tex”. Then the compilation generates main.bbl and main.aux among several others. If you keep getting errors when you know the syntax is correct then try:

rm main.aux main.bbl

pdflatex main
bibtex main
pdflatex main
pdflatex main

and see if that clears the error.

Saving / resurrecting old Fortran programs

A general rule for resurrecting old code in any language is first try to get it working for known input/output via emulation or virtual machine. Many old codes were made before linters and other correctness checks we’ve long taken for granted. The codes may use non-standard tricks no one has thought about for years to squeeze tiny fractions of efficiency that haven’t been significant for decades.

After correct operation is established, Fortran 66 code often needs a few tweaks to work with modern compilers. The Fortran Wiki has an excellent article on modernizing Fortran syntax http://fortranwiki.org/fortran/show/Modernizing+Old+Fortran. Here are a few specific techniques we use.

Regex operations

Obviously make a copy of the file first. Sometimes 10,000+ lines will match so it can be tedious to check. Similar operations may be needed if a mix of tabs and spaces are used.

move line numbers left

Run this regex and replace with null (nothing):

^\s+(?=\d+\s)

make code start in column 7

Recall that fixed format (Fortran <= 77) code has line numbers in columns 1-5 of each line numbers After the prior operation, push the actual code right of column 6 by replacing this regex with appropriate number of spaces:

(?<=^\d+\s+)

multiline codes

You may see $ used like a semicolon in CDC Cyber Fortran

Duplicated line numbers

Put procedures in separate files for duplicated line numbers.

Delete Git tag / release version

Deleting or updating a GitHub or GitLab release leaves the Git tag, which must also be deleted to reissue a corrected release.

  1. delete the Git tag locally. Here we assume the release tag was v1.2.3

    git tag -d v1.2.3
    
  2. delete the Git tag on remote site

    git push -d origin v1.2.3
    

    This puts the release notes from v1.2.3 into “draft” mode–hidden from the public.

  3. Make the necessary changes, then do the release again, reusing the draft release notes.

Notes

FFmpeg setting to upload YouTube video

If needing to re-encode video anyway, for example if coming from an a scientific camera capture with uncompressed video, the suggested settings for YouTube upload using FFmpeg are:

ffmpeg -i in.avi -c:v libx264 -preset slow -crf 18 -c:a aac -b:a 192k -pix_fmt yuv420p -movflags +faststart out.mp4

If the video has no audio, use:

ffmpeg -i in.avi -c:v libx264 -preset slow -crf 18 -pix_fmt yuv420p -movflags +faststart out.mp4

Notes

FFmpeg for YouTube

Python zipfile recursive write

This function recursively zips files in directories, storing only the relative path due to the use of the “arcname” parameter. Otherwise, you get the path relative to the root of your filesystem, which is rarely useful.

import zipfile
from pathlib import Path


def zip_dirs(path: Path, pattern: str) -> T.List[Path]:
    """
    recursively .zip a directory
    """

    path = Path(path).expanduser().resolve()

    dlist = [d for d in path.glob(pattern) if d.is_dir()]
    if len(dlist) == 0:
        raise FileNotFoundError(f"no directories to zip under {path} with {pattern}")

    for d in dlist:
        zip_name = d.with_suffix(".zip")
        with zipfile.ZipFile(zip_name, mode="w", compression=zipfile.ZIP_LZMA) as z:
            for root, _, files in os.walk(d):
                for file in files:
                    fn = Path(root, file)
                    afn = fn.relative_to(path)
                    z.write(fn, arcname=afn)
        print("write", zip_name)

Matlab parfor parallel plotting

Matlab plotting can be quite slow, as can Python Matplotlib plotting. Sometimes, Matlab parfor can be used to plot in parallel, when all parfor restrictions are met. However, parallel plotting in Matlab doesn’t always work, or may work on some operating systems but not others. So use great caution if making a “parfor” plotting loop–it may not work for others.

Errors noted

Here are some errors we’ve gotten when trying to do relatively simple plots in parallel:

Warning: A worker aborted during execution of the parfor loop. The parfor loop will now run again
on the remaining workers.
> In distcomp/remoteparfor/handleIntervalErrorResult (line 240)
  In distcomp/remoteparfor/getCompleteIntervals (line 387)
  In parallel_function>distributed_execution (line 741)
  In parallel_function (line 573)
Error using distcomp.remoteparfor/rebuildParforController (line 194)
All workers aborted during execution of the parfor loop.

Error in distcomp.remoteparfor/handleIntervalErrorResult (line 253)
                obj.rebuildParforController();

Error in distcomp.remoteparfor/getCompleteIntervals (line 387)
                            [r, err] = obj.handleIntervalErrorResult(r);
Warning: worker(s) crashed while executing code in the current parallel pool. MATLAB may attempt
to run the code again on the remaining workers of the pool, unless an spmd block has run. View the
crash dump files to determine what caused the workers to crash.

Matlab system stdin pipe

Matlab does not have a native means to pass Matlab data via a stdin pipe to programs called with system. This is not an issue with Python subprocess that is quite robust and featureful. “stdout” and “stderr” are joined by Matlab, available at program completion like:

[ret, msg] = system(...);

where “msg” is the data from stdout and stderr joined together.

The benefits of this method is that is can be faster than using temporary files, and helps avoid filesystem clashes when running many external processes in parallel or asynchronously. This method avoids the need to write additional code directly interfacing memory between Fortran or C and Matlab by using file-based or pipe-based API for data streaming. The penalty for this simplicity is execution speed, but that can be resolved later if I/O speed is a genuine bottleneck by using Matlab Engine or MEX.

Example

As noted by Mathworks staff, Python or Java can be used to pass stdin from Matlab to executable.

These examples can readily be expanded to handle arbitrarily large and complex inputs and outputs via stdin and stdout command line pipes.

Matlab stdin pipe examples (GitHub)

GNU Octave

It is also possible to do this via Java in Octave, but the syntax is slightly different using JavaObject, which is left as an exercise for the reader.

Fix Python tempfile PermissionError on Windows

Python tempfile is normally robust but has some corner cases on Windows. A common issue is that on exiting a tempfile context manager, PermissionError is thrown on Windows. This can happen when non-writable files are placed into a tempfile.TemporaryDirectory such as cloning a Git directory into the TemporaryDirectory.

Workaround

A common workaround is to simply not use the tempfile context manager, instead manually deleting the temporary directory when done:

from pathlib import Path
import tempfile

temp_dir = tempfile.TemporaryDirectory()
temp_path = Path(temp_dir.name)

# do stuff in temporary directory

# when done with temporary directory
try:
    temp_dir.cleanup()
except PermissionError:
    pass

If there is a PermissionError, the temporary directory remains until the operating system cleans up the temporary directory.

Matlab / Octave detect JVM

Matlab “.m” script functionality can be readily extended with Java and Python code. It may seem obvious to check if Python is available. It is also important to check if the Matlab JVM interface is available as Matlab may be running in -nojvm mode. Starting up Matlab without the JVM disables plotting and Java functions and can be as much as 10x faster to startup Matlab:

matlab -nojvm

Detect if JVM is available from within a Matlab script by:

hasjvm = usejava('jvm');
% boolean

GNU Octave

Octave also has a JVM interface that extends Octave functionality. As with Matlab, the Octave JVM interface availability is checked by

hasjvm = usejava('jvm');
% boolean

Change file ownership

When desired and permitted by the computer filesystem, it’s possible to change file ownership. This may be necessary when a file was inadvertently created with root / admin ownership by mistake, and it’s necessary for a general user to edit or access the file. Changing file ownership can have unexpected consequences, like removing the ability of others to access the file or run a program depending on the file. Therefore, file ownership changes should be done only when necessary and with consideration for others who may depend on the file.

Examples

These examples assume the file “my.txt” is in the current directory and the user logged in should own the file. These tasks are similarly done in other languages such as:

Python

As long as the user has filesystem permission, Python can easily change file ownership across operating systems.

import shutil
import getpass

shutil.chown("my.txt", user=getpass.getuser())

docs

Windows

Check ownership of the file by

  • Command Prompt

    dir /q my.txt
    
  • PowerShell

    (get-acl my.txt).owner
    

Windows uses the takeown command to change file ownership. For simplicity we assume the desired user is logged in and executing the command.

takeown /f my.txt

MacOS / Linux

Check ownership of the file by

ls -l my.txt

For simplicity we assume the desired user is logged in and executing the command chown:

chown $whoami:$whoami my.txt