rclone for Google Drive and Dropbox

rclone seamlessly connects to remote cloud storage on Mac, Linux, Windows and several other operating systems for cloud storage services including Google Shared Drive, Dropbox, Microsoft OneDrive and many more providers. The rclone CLI is easier than rsync. rclone works with encrypted filesystems.

Setup Rclone:

  • Linux: Download and extract–for PC, use amd64. Add to “~/.bash_aliases”:

    alias rclone='~/rclone/rclone'

  • MacOS / Homebrew: brew install rclone

  • Windows: Download, extract, install

Add various remote file shares (Google Drive, Dropbox, etc.) by:

rclone config

In these examples, remote is the particular drive name chosen during rclone config.

Configuration is stored in ~/.config/rclone/rclone.conf.

List remote directories:

rclone lsd remote:

List remote directories under path/to

rclone lsd remote:path/to

List remote files under path/to

rclone ls remote:path/to

Count number of files in path/to

rclone ls remote:path/to | wc -l

copy in rclone by default does not clobber existing remote files, if they haven’t been changed.

Recursively copy local computer ~/path to remote

rclone copy ~/path remote:path -v

Copy only this directory contents ~/path to remote

rclone copy --max-depth 0  ~/path remote:path -v

Using rclone sync requires caution, because sync DELETES files on remote that are not also on local PC!

Sync local computer ~/path to the remote

rclone sync ~/path remote:path -v

Linux swap file

If too much swap space is filled, the system could become unusable until force-rebooted due to very slow hard drive (or SSD) I/O relative to RAM. It can be better to have the application terminated automatically due to running out of RAM instead of having the system become inaccessible until force-rebooted. There are disadvantages for no swap file.

If not using hibernation, 100 MB swap size is a general starting point for laptop, desktop and embedded systems. Consider the specific situation and test before deployment to avoid unexpected impacts. Setting swapfile size is only for systems using a swapfile. Systems using a swap partition require additional knowledge.

For systems using Dphys-swapfile such as Raspberry Pi, check swap status:

systemctl status dphys-swapfile

Persistently disable swap by:

systemctl stop dphys-swapfile
systemctl disable dphys-swapfile

CMake TLS VERIFY global

TLS verification defaults to OFF in CMake, yet there is a significant security benefit from globally setting TLS verification ON. This is accomplished via CMake variable CMAKE_TLS_VERIFY.

CMAKE_TLS_VERIFY allows a user to globally configure certificate verification. TLS verification can be an important part of cybersecurity. It’s often better to not have to worry about commands missing this parameter–just set it once. In case of suspected broken certificates, verification can then be easily switched off temporarily.

We suggest near the beginning of the CMake project:

set(CMAKE_TLS_VERIFY true)

Example

Here we use badssl.com, that purposefully had a variety of certificate problem URLs.

cmake_minimum_required(VERSION 3.19)

set(CMAKE_TLS_VERIFY ON)

# Get CMake's vendored cURL version
file(DOWNLOAD https://www.whatsmyua.info/api/v1/ua ua.json)
file(READ ua.json meta)
string(JSON ua GET ${meta} 0 ua rawUa)

message(STATUS "CMake ${CMAKE_VERSION}
cURL version: ${ua}
TLS_CAINFO: ${CMAKE_TLS_CAINFO}
SSL_CERT_FILE: $ENV{SSL_CERT_FILE}"
)


function(ssl url)

file(DOWNLOAD ${url} STATUS ret)

list(GET ret 0 status)
list(GET ret 1 msg)

message(STATUS "${url}:${status}: ${msg}")

endfunction(ssl)


message(STATUS "These tests should fail.")
ssl(https://expired.badssl.com/)
ssl(https://untrusted-root.badssl.com/)

message(STATUS "These tests should work OK.")
ssl(https://hsts.badssl.com/)
ssl(https://sha256.badssl.com/)

Matlab system() environment variables

Matlab system() discards the shell environment variables. To set environment variables, set them in name-value pairs like:

if ispc
  system("echo %test_var%", test_var="hello")
else
  system("echo $test_var", test_var="hello")
end

Multiple environment variables can be set in a single command by adding more name-value pairs.

If there are known environment variables desired to set in the system command, do like:

system("make", CC=getenv("CC"))

If there is a struct of environment variables, do like:

envStruct = struct("CC", "gcc", "CXX", "g++");

envCell = namedargs2cell(envStruct);

system("make", envCell{:});

CMake environment variable scope

Environment variables in CMake set outside of CMake before a CMake process is started have global scope in CMake. Setting environment variables in a CMake script don’t have visibility outside the current CMake process. That includes CMake ExternalProject, which will NOT see environment variables set(ENV{name}) in the parent CMake project. Also, build and test processes will NOT see environment variables set in the CMake script.

To make environment variables take effect in a CMake script or CMake configure, set the environment variable before running CMake, or on a Unix-like OS on the command line. For example, to tell CMake the HDF5 library location, set environment variable HDF5_ROOT before the first CMake configure command:

# Unix-like
HDF5_ROOT=~/mylib cmake -Bbuild

# Windows PowerShell
$env:HDF5_ROOT="~/mylib"
cmake -B build

Of course, one could also set the CMake variable at configure like:

cmake -Bbuild -DHDF5_ROOT=~/mylib

To make environment variables present during a CMake build, for example for an ExternalProject, do as above for the configure.

To control environment variables present for CTest tests, including for individual tests, set test properties.

Fortran filename suffix

For modern Fortran development, “.f90” is the recommended filename suffix. For modern Fortran code that needs to be preprocessed (e.g. with #ifdef statements) then “.F90” is the recommended filename suffix.

For legacy Fortran 77 code, “.f” is the recommended filename suffix. For Fortran 77 code that needs to be preprocessed, “.F” is the recommended filename suffix.

For a time some had suggested other filename extensions, but being as Fortran standard only recommends the latest version, by default we use the first “free-form” syntax Fortran 90 standard “.f90” suffix to indicate the current Fortran standard. This helps avoid too many ambiguous file suffixes.

C and C++ likewise do not indicate their language standard year in the file suffix.

CMake ExternalProject ensure same compilers

Common among build systems is the use of environment variables CC, CXX, FC to signal user intent to use a compiler, regardless of what appears first in environment variable PATH. A contradiction can arise for a CMake project using ExternalProject in that CMAKE_C_COMPILER et al are not automatically passed to the ExternalProject. Thus if environment variable “CC=gcc” but the top-level project user specified “cmake -DCMAKE_C_COMPILER=icx”, the top level CMake project would use icx IntelLLVM but the subproject would use GCC, which can cause unintended results.

The fix for this issue is to explicitly pass CMAKE_C_COMPILER et al to the ExternalProject CMAKE_ARGS if the subproject is also a CMake project.

set(args
-DCMAKE_C_COMPILER=${CMAKE_C_COMPILER}
-DCMAKE_CXX_COMPILER=${CMAKE_CXX_COMPILER}
-DCMAKE_Fortran_COMPILER=${CMAKE_Fortran_COMPILER}
)

ExternalProject_Add(...
CMAKE_ARGS ${args}
)

For Autotools ExternalProject do like:

set(args
CC=${CMAKE_C_COMPILER}
CXX=${CMAKE_CXX_COMPILER}
FC=${CMAKE_Fortran_COMPILER}
)

ExternalProject_Add(...
CONFIGURE_COMMAND <SOURCE_DIR>/configure ${args}
)

For GNU Make subproject do like:

set(args
CC=${CMAKE_C_COMPILER}
CXX=${CMAKE_CXX_COMPILER}
FC=${CMAKE_Fortran_COMPILER}
)

ExternalProject_Add(...
CONFIGURE_COMMAND ""
BUILD_COMMAND ${MAKE_EXECUTABLE} -j ${args}
)

CMake archive extract syntax

CMake 3.18 added file(ARCHIVE_EXTRACT), which is much more robust and easy to use than the prior syntax.

# recommended
file(ARCHIVE_EXTRACT ${archive} ${out_dir})

# older, less robust
file(MAKE_DIRECTORY ${out_dir})

execute_process(
COMMAND ${CMAKE_COMMAND} -E tar xf ${archive}
WORKING_DIRECTORY ${out_dir}
RESULT_VARIABLE ret
)
if(NOT ret EQUAL 0)
  message(FATAL_ERROR "extract ${archive} => ${out_dir}    ${ret}")
endif()