SCP/SSHFS recursive copy with exclusions
SCP does not have an option to exclude files while copying remote files over SSH. This is a problem when you have Git-managed code you’ve modified on say:
- offline computer
- HPC, and don’t want to put your Git host credentials on the HPC
If you just use scp -r
, you’ll also overwrite the .git
directory which can destroy work done on this or other branches.
We want to just copy the code files, NOT the .git
tree.
Solution
on your laptop, install SSHFS, which uses SSH and FUSE to make a virtual directory on your laptop, where the true files reside on the remote computer.
- Linux / Windows Subsystem for Linux:
apt install sshfs
- Mac:
brew install sshfs
- Linux / Windows Subsystem for Linux:
mount to a location on your laptop, say
~/X
mkdir ~/X # one-time sshfs -o follow_symlinks myserver: ~/X
``
Now the
~
directory onmyserver
is connected to~/X
on your laptop.use
cp
as usual to copy a directory while excluding.git
shopt -s extglob # enables fancy globbing cp -rv ~/X/myprog/*!(.git/|bin/) .
``
Note I also excluded things under
bin/
of the repository, as the HPC probably runs a different Linux distro and the compilation is optimized for a different CPU than my laptop, so the HPC binaries wouldn’t generally be useful on my laptop.