This isn’t part of rsbackup proper, but since you
might well want to encrypt your backups, the setup process is
sketched here.
The main threat I’m interested in protecting against is that a petty criminal comes into possession of my backup disk either by stealing it from an offsite location, or because I manage to lose it somehow. I don’t want them able to read my email or gain access to website passwords (of which I have far too many to remember, and most of them don’t protect anything especially important).
I’m not especially interested in protecting my backups from the NSA. If an organisation with government-level resources were to take an interest in me, then realistically they have better tools available than stealing my backups.
The reason I document this is that, if you are interested in defending against better-resourced attacks than I am (for instance if you live in a country with a highly repressive government), the details below may not be suitable for you.
The device mapper is the Linux kernel’s framework for creating virtual block devices. In this case, we are interested in creating a virtual block device that has the plain text corresponding to encrypted data on a physical block device.
dm-crypt is the low-level means of achieving this: writes to the virtual block device are encrypted and forwarded to the underlying physical device, and similarly reads from the virtual block devices are forwarded to the physical device and the content decrypted. The plain text will not be available if the correct key has not been supplied.
Linux Unified Key Setup (LUKS) is a specific encryption scheme that we’ll use.
cryptsetup is a command line tool we’ll use to set things up.
First create the partition to encrypt, using your favourite
disk partitioner, lvcreate, or whatever. Any
pre-existing contents will be destroyed, so take a backup if
there is anything important there. I’ll assume below that the
device name for the partition it /dev/sdb1, but
obviously you should change this to whatever device you are
using.
To create the LUKS data structures and establish a key:
# cryptsetup luksFormat /dev/sdb1 WARNING! ======== This will overwrite data on /dev/sdb1 irrevocably. Are you sure? (Type uppercase yes): YES Enter LUKS passphrase: Verify passphrase: Command successful.
As you can see, you must choose a passphrase. The default encryption key is 128 bits long, so it’s worth using a longer password than the traditional 8 characters, provided you can actually remember it. (A password you can’t remember is no use whatsoever.)
Note: cryptsetup also supports reading the key
from a file. If keeping the key file safe somewhere (a couple
of well-hidden USB sticks, say) is easier than remembering a
suitably long passphrase, that might be more appropriate.
Note: cryptsetup offers a variety of
cipher specifications. You may wish to review the available
options and consult the cryptsetup
FAQ rather than accepting the default.
At this point the encrypted partition exists but does not have any filesystem in it and the underlying plaintext is not accessible. It’s possible to detect the format:
# cat /dev/sdb1 | file - /dev/stdin: LUKS encrypted file, ver 1 [aes, cbc-essiv:sha256, sha1] UUID: c3ad50a5-a379-4e72-9f92-cacf592
The next step is to create a virtual block device with the plaintext:
# cryptsetup luksOpen /dev/sdb1 backup3 Enter LUKS passphrase: key slot 0 unlocked. Command successful. # ls -l /dev/mapper/backup3 brw-rw---- 1 root disk 254, 7 2010-03-14 15:54 /dev/mapper/backup3
You will need to re-enter the passphrase you chose earlier.
At this point you can create a filesystem:
# mkfs -j -Lbackup3 /dev/mapper/backup3
mke2fs 1.41.3 (12-Oct-2008)
Filesystem label=backup3
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
62336 inodes, 248870 blocks
12443 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=255852544
8 block groups
32768 blocks per group, 32768 fragments per group
7792 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376
Writing inode tables: done
Writing superblocks and filesystem accounting information: done
This filesystem will be automatically checked every 29 mounts or
180 days, whichever comes first. Use tune2fs -c or -i to override.
When I did this on a 4TB USB2-attached disk, it took about 40 minutes. So you might want to go away and do something else.
By default the filesystem will be regularly fsck’d. You can suppress this, if you want:
# tune2fs -c0 -i0 /dev/mapper/backup3
It’s now possible to mount the new filesystem:
# mount /dev/mapper/backup3 /mnt # really ls -l /mnt total 16 drwx------ 2 root root 16384 2010-03-14 15:55 lost+found richard@araminta:~$ df -h /mnt Filesystem Size Used Avail Use% Mounted on /dev/mapper/backup3 957M 1.2M 908M 1% /mnt
So now you can create files, take backups, etc.
Note that just because the disk is encrypted does not imply that other users of the system cannot get at its contents while it’s mounted. The normal file permission rules apply.
To unmount and detach the disk:
# umount /mnt # cryptsetup luksClose backup3
The Debian package of rsbackup has some extra
features. You could use these on other systems but on Debian they
are set up to work automatically.
/etc/rsbackup/configIn the .deb package, the default
/etc/rsbackup/config is as follows:
# Location of lockfile lock /var/run/rsbackup.lock # User configuration include /etc/rsbackup/local # Hosts include /etc/rsbackup/hosts.d
Normally you would not edit this file. Instead, put local
configuration in /etc/rsbackup/local and a file for
each host to back up into /etc/rsbackup/hosts.d.
/etc/rsbackup/defaultsThere is an extra configuration file for the cron jobs,
/etc/rsbackup/defaults. The default is as
follows:
# # Set backup=hourly|daily|weekly|monthly to control frequency of # backup attempts. (Use backup policies for fine-grained control over # when backups happen.) # backup=hourly # # Set report=hourly|daily|weekly|monthly to control frequency of # email reports. (Hourly is probably a bit much!) Only effective # if email is not "". # report=daily # # Set email=ADDRESS to have the report emailed to that address. # email=root # # Set prune=hourly|daily|weekly|monthly|never to control frequency of # automated pruning of old backups # prune=daily # # Set prune_incomplete=hourly|daily|weekly|monthly|never to control # frequency of automated pruning of incomplete backups # prune_incomplete=weekly # # Prefix to the rsbackup command # Use 'nice' and/or 'ionice' here # nicely=
Set backup to control the frequency of backup
attempts. Normally this can be left at hourly and
the frequency of each host and volume’s backups controlled via
backup policies.
Set email to the destination address for
reports from the daily backup run, or comment out the line
entirely to suppress email reports. If you want reports at
some other frequency than daily, modify
report.
Modify prune to control the frequency of
pruning of old backups. It is recommended to leave this as
daily, since deleting a week or more’s worth of
backups takes a very long time.
Modify prune_incomplete to control the
frequency of deleting incomplete backups. It is recommended
to keep this at a lower frequency than you take backups, as
otherwise rsync will not be able to use
incomplete backups to optimize new ones.
Modify nicely to control the priority of
rsbackup. For example, you might use:
nicely="nice ionice -3"
1. Overview
2. Setup
2.1 Installation
2.2 Configuration
3. Manual Backups
3.1 Initial Backup
3.2 Pruning Old Backups
3.3 Failed Backups
3.4 Retired Volumes
3.5 Status Reports
4. Automatic Backups
5. Snapshots
6. Device Management
6.1 Lost Devices
6.2 Upgrading Devices
7. Restoring
7.1 Manual Restoration
7.2 Restoring With rsync
8. Links
rsbackup backs up your computer(s) to removable
hard disks. The backup is an ordinary filesystem tree, and hard
links between repeated backups are used to save space. Old
backups are automatically pruned after a set period of
time.
This guide describes how to set up and manage
rsbackup. See the man pages for the command and the configuration file for detailed
reference information.
The systems you want to back up are called clients.
The system that has the backup hard disk(s) attached to it is
called the server, and it is on this system that the
rsbackup program runs. The server can itself be a
client.
Each client must have an SSH server and rsync installed. For Debian and Ubuntu systems it should be sufficient to install them as follows (if you don’t have them already):
apt-get install openssh-server rsync
The server requires rsync and an SSH client. Again for Debian:
apt-get install openssh-client rsync
For other platforms, you must consult their documentation, or install them from source:
The server’s root login needs to be able to SSH to each of the clients’ root logins without having to enter a password or confirm a key hash. You should consult the SSH documentation to set this up, but the general procedure, assuming you use RSA keys and OpenSSH, is as follows. If you are sufficiently familiar with SSH to do this without further documentation, skip to the next section.
On the server create an SSH key with:
sudo ssh-keygen
When asked for a passphrase, just hit return (but see
below). Then copy ~root/.ssh/id_rsa.pub to each
of the clients and append it to their
~root/.ssh/authorized_keys. At the same time,
retrieve the clients’ host key hashes with:
ssh-keygen -l -f /etc/ssh/ssh_host_rsa_key.pub
As root on the server, ssh to each of the clients
and verify their host keys hashes.
rsbackupTo install rsbackup, go to www.greenend.org.uk/rjk/rsbackup
and download the source code:
wget http://www.greenend.org.uk/rjk/rsbackup/rsbackup-_version_.tar.gz tar xf rsbackup-_version_.tar.gz cd rsbackup-_version_ ./configure make sudo make install
(You will probably need to change the version number.)
On Debian systems you can use the pre-built
.deb files:
wget http://www.greenend.org.uk/rjk/rsbackup/rsbackup__version__amd64.deb sudo dpkg -i rsbackup__version__amd64.deb
(You will probably need to change the version number and perhaps the architecture.)
At this point it should be possible to read the man page, which contains reference information:
man rsbackup
What If The Server Is Also A Client?
If you want to backup the backup server itself then you don’t need to set up the server to be able to SSH to itself. See below for how to configure this.
What If I’m Not The Superuser?
rsbackup does not actually depend on being
the superuser, although of course its functionality will
be limited if it isn’t. However you could for instance
use it to back up your home directory to your portable USB
disk. The setup is the same except that you do it for
your personal login rather than root and therefore don’t
use sudo.
What If I Don’t Like Empty Passphrases?
In this case you will have to find some other way of making the server’s private SSH key available when backups run. This is outside the scope of this document.
rsbackup reads a configuration file from
/etc/rsbackup/config. (Use the
--config option to override this, if you prefer
another location.) You will need to enter some global
settings and then describe the backup clients.
First you should define where backups will be stored. This guide will assume you use removable hard disks, but you can use permanently online backups too.
For each distinct backup device you need to define two
things. The first is the mount point that the device will
appear at. For example, if you have two backup disks and the
mount points are /backup1
and /backup2 you would write the following:
store /backup1 store /backup2
The second is to define device names. Device names
correspond to the contents of a single-line file
called device-id in the root of the backup device.
For instance, if you called your devices backup1
and backup2 you would write the following:
device backup1 device backup2
Of course, you must also create these files! For example:
echo backup1 > /backup1/device-id echo backup2 > /backup2/device-id
rsbackup does not mind if the devices share a
mount point (and only one is present at a time); any device
may use any mount point as far as it’s concerned. You will
probably find it more convenient to give them separate mount
points though. If more than one device is mounted when you
make a backup, backups will be made to all of them.
You can use the --store option to select just
one.
Although it would of course be convenient for users to be able
to access backups of their files directly, it would also mean
that they could go “back in time” past permission changes or
deletions of private files belonging to one another.
Therefore, the top directory of your backup devices should
(usually) be owned by root and mode 0700
(i.e. -rwx------).
By default, rsbackup will insist on this,
although you can use the public directive to change
this behaviour.
chmod 700 /backup1 /backup2
Remember to update /etc/updatedb.conf to
exclude your backup devices. Otherwise updatedb
will spend ever-increasing amounts of time indexing your
backups.
Next you may want to define some pruning parameters. These can be overridden for each volume you back up, so by defining global ones you are only setting defaults.
The first one you might want to set is the maximum age of the most recent backup. If any volume’s most recent backup is older than this many days then it will show up as red in the backup status report. The default is 3 days. To reduce it to (for instance) 1 day:
max-age 1
The second parameter to choose is the age at which backups are automatically pruned, i.e. deleted. The default is 366 days, ensuring that you will be able to “go back” up to a year. If you only wanted to go back a month you could reduce it as follows:
prune-parameter prune-age 31
Remember, these are defaults and can be overridden on a per-host or per-volume basis.
There are many other settings described in the man page. They will not be covered here.
The rest of the configuration file will define what to back up (and what to exclude).
There are two ways to organise your configuration file.
You can put all the configuration for all the hosts in the main configuration file.
You can put each host’s definitions in a file of its own and include them all. To do this put a line at the end of your configuration file as follows
include /etc/rsbackup/hosts.d
Then for each host create a file named after the host in this directory and use it to store the host’s configuration, as described below.
The the Debian packaging of rsbackup uses this approach.
For each host to back up, you should write a host stanza. This will contain some host level settings and then a volume stanza for each part of the host’s filesystem back up.
Here is an example host stanza:
host sfere
volume root /
volume boot /boot
volume home /home
prune-parameter prune-age 366
exclude /*/Desktop/Trash/*
exclude /*/.local/share/Trash/*
exclude /*/.mozilla/*/Cache/*
exclude /*/.ccache
volume var /var
exclude /tmp/*
The meaning of this is as follows:
The first line contains the name of the host. This would normally be its DNS name (see below for an example of where it is not).
Each of the volume lines contains the name of
a volume on the host and the path to that volume. By
default, rsbackup will assume that each volume
corresponds to a (mounted) filesystem and therefore not backup
files from other filesystems.
In this case there are four volumes. root
and boot are quite simple: all the files in them
will be backed up.
home, however, is more complex. Firstly, it
has a prune-age setting to ensure that it is kept
for longer than the default lifetime. Secondly, it excludes
various trash and cache directories.
In the first three cases, it does this by backing up the
directory but not its contents; they will be empty on the backup
device. In the fourth case, it does not even backup the
directory. Note that the exclusion patterns are rooted at the
path to the volume - they are not absolute path names.
(Consult the rsync documentation for --exclude for
more information about these patterns.)
/var is similar to home in that
a temporary directory is excluded.
An important note: the indentation is not
significant to rsbackup - only to the reader.
Anything that comes after a host directive and
before the next host or volume
directive is considered part of that host. Similarly anything
that comes after a volume directive and before the
next host or volume directive is
considered part of that volume.
This example shows how to back up a host where the name
differs from the DNS name. The important part is the
hostname directive:
host lithhostname chymax volume lith /Volumes/Lith exclude "Temporary Internet Files" exclude /RECYCLER exclude /pagefile.sys exclude "/Documents and Settings/*/Local Settings/Temp" exclude "/System Volume Information/_restore*"
What is actually going on here is that lith is
really the Windows partition on chymax. The
computer usually runs Unix, with the Windows partition mounted
for convenience. So to get lith’s files, it is
necessary to ssh to chymax.
This example shows how to back up without using SSH at all:
host araminta
hostname localhost
volume root /
volume boot /boot
volume home /home
prune-parameter prune-age 366
exclude /*/Desktop/Trash/*
exclude /*/.local/share/Trash/*
exclude /*/.mozilla/*/Cache/*
exclude /*/.ccache
volume var /var
exclude /tmp/*
volume news /var/spool/news
prune-parameter prune-age 14
In this case, araminta is actually the backup
server, so using SSH would mean SSHing to itself. The hostname
localhost is special-cased to avoid using SSH at all.
Here is example of backing up a laptop:
host kakajou
max-age 7
volume users /Users
prune-parameter prune-age 366
exclude /*/.Trash/*
exclude /*/Library/Caches
exclude /*/Library/VirtualBox/Machines/*/Snapshots
exclude /*/Library/VirtualBox/**.vdi
volume local /local
prune-parameter prune-age 366
volume etc /etc
volume sw /sw
This host is usually asleep or not even in the house, so
opportunities to back it up are rare. Therefore it has a
host-wide max-age setting.
The version of rsync supplied by Apple does
not support the options used by modern versions for backing up
extended attributes and ACLs. If you have not installed a
modern version of rsync then you must
configure rsbackup to use suitable options
instead:
rsync‐extra‐options ‐‐extended‐attributes
Note that this will not back up ACLs.
This is only suitable for local backups. For remote backups
the situation is worse still. If backing up a macOS host from
a host with a modern
rsync, or vice versa, extended attributes and ACLs
cannot be backed up at all.
The directive must then be set as follows:
rsync-extra-options
Note that this will not back up ACLs or extended attributes.
When backing up a Windows filesystem it can happen that attributes in the Windows filesystem do not fit in the backup filesystem; if this happens you may see errors like this:
rsync: rsync_xal_set: lsetxattr(""/backup7/host/volume/2018-02-04/path/to/file"","attrname") failed: No space left on device (28)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.2]
In that case the affected volumes must disable attribute backup and ACL backup as follows:
rsync-extra-options
Note that this will not back up ACLs or extended attributes.
Before you actually make a backup, you should do a “dry run”
to verify that rsbackup does what you expect.
rsbackup --backup --dry-run
This will print out the commands that it would run without actually executing them. It’s also a good way of verifying that the syntax of the configuration is correct.
Once you’re happy with the output, you can try making an initial backup:
rsbackup --backup --verbose
The --verbose option makes rsbackup
report what it is doing as it progresses. In normal use you
would omit it but it’s useful when setting up and crucial when
debugging.
Depending on how much data you have (and how fast your disks and network are) the initial backup may take a very long time. I did my initial backups inside an instance of screen so that they couldn’t be affected by logging out, etc.
For each backup of each volume, an entry will be made in
/var/log/backup/backups.db detailing what was
backed up and recording
any error output.
Pruning refers to deleting a volume’s old backups. During
pruning, a backup will be deleted if it is older than the
volume’s max-age setting, with the proviso that the
most recent min-backups backups on each store will
not be deleted.
To prune any backups that are due to be deleted:
rsbackup --prune --verbose
The details of what is pruned are logged to files
in /var/log/backup.
There are several possible pruning policies, which determine which backups are selected for deletion. See the section on pruning in rsbackup(5) for details.
If a backup fails then it will be left in an incomplete
state. You can tell rsbackup to pick up where it
left off simply by running it again on the same day; if however
you leave it until another day then that backup will never be
completed. To delete any incomplete backups, therefore:
rsbackup --prune-incomplete --verbose
Backups are not pruned immediately because even if they are incomplete, the portion that succeeded may be useful to reduce the amount of data copied when you retry.
When you take a volume or host out of service, you need to
tell rsbackup about this. The first part of this
is to remove the corresponding volume or
host sections in the configuration file. If you
don’t do this then rsbackup will keep on trying
to backup the obsolete volume or host.
The second part is to delete old backups for the volume and
their corresponding records. If you don’t do this then the
report will complain about them. This can be done with the
--retire option:
rsbackup --retire HOST:VOLUME
...or:
rsbackup --retire HOST
All backups on available devices and their corresponding records will be deleted (possibly a slow process).
If you want to keep (say) the last backup for the volume
then you should at least rename it aside, otherwise
--retire will delete it; you may prefer to tar it
up and compress it.
It is also possible to keep the backups themselves, but
remove the record of them, making them invisible to
rsbackup but preserving the data for archival. To
do this, use --forget-only together with the
--retire option.
You can generate an HTML status report to a disk file:
rsbackup --html status.html
This will show:
max-age setting for the volume and the latter is
red if there are no backups of the volume on the device.rsync output for the last backup, if it failed.
You can
use the --logs option to make the logfile section
more verbose.If you prefer plain text you can use --text
instead of --html.
You can also request that the report by sent by email, with
the --email option. This is intended for use
when automating backups. The email contains both the plain
text and the HTML versions, most mail clients should be able
to pick whichever they are best able to display.
If you installed from a .deb then have a look at
the Debian-specific documentation.
Manual backups might be perfectly adequate for you. However, computers are often better at remembering to perform scheduled tasks than humans are, so it may be better to run your backups as a cron job. For example, to run your backups, with pruning and a report, at 1am every day you might use the following crontab line:
0 1 * * * rsbackup --backup --prune --email root
This will automatically do a backup every night, prune any out-of-date backups, and mail a report to the system administrator.
You might want to add (for example) a weekly prune of incomplete backups:
0 8 * * 0 rsbackup --prune-incomplete
If you use Linux LVM then you may prefer to snapshot filesystems while they are being backed up, so that there is no possibility of files changing half way through the backup. This can be achieved by adding the following lines to your configuration file:
pre-volume-hook rsbackup-snapshot-hook post-volume-hook rsbackup-snapshot-hook
Then for each volume that is to be snapshotted, create
/snap/VOLUME on the target host. The volume
name must be the one used by rsbackup, for instance
root, boot, home or
var in the first example
above. Before each volume is backed up, a snapshot will
automatically be created; it will be removed again after the
backup is complete or on failure.
There is a man page for rsbackup-snapshot-hook detailing what it does and the options it can take.
Hard disks get lost or stolen, and fail. In this
case rsbackup needs to be told that one of its
devices has gone away. The first part of this is to delete the
corresponding device directive in the configuration
file (and the store directive, if that’s unique to
the device).
The second part is to delete records for backups to the
device. If you don’t do this then the report will complain
about them. This can be done with the
--retire-device option:
rsbackup --retire-device NAME
You can do these steps in either order, but if you delete
the records first, you will be ask if you are sure. You can
override this with the --force option.
--retire-device will never delete actual
backups, only the backup records.
If a backup device gets full you have several options:
Reduce the number of backups to be kept on it by
lowering prune-age. But sooner or later you
may reach the point where you just cannot keep backups as
long as you like.
Introduce an entirely new, bigger device and take the old device out of service, either keeping it against a rainy day or destroying it as described above.
Copy all its contents to a new, bigger device,
keeping the same device name. Remember to delete the old
device-id file, or confusion may
follow!
Backups aren’t worth anything if you can’t restore, course.
The backup has the same layout, permissions etc as the original system, so it’s perfectly possible to simply copy files from a backup directory to their proper location. This is the most convenient way if you want to rescue only a small number of files.
Be careful to get file ownership right. The backup is stored with the same numeric user and group ID as the original system used. Put another way, the relationship between usernames and user IDs (and group names and group IDs) on the backup disk reflects the client, not the server (or any other machine the disk might be attached to).
Until a backup is completed, a corresponding
.incomplete file will exist. Check for such a
file before restoring any given backup.
rsyncYou can do bulk restores with rsync. For
example, supposing that host chymax has a volume
called users in which user home directories are
backed up, and user rjk wants their entire home
directory to be restored:
rsync -aSHAXz --numeric-ids /store/chymax/users/2010-04-01/rjk/. chymax:~rjk/.
-a means recursive into directories; preserve
symlinks, permissions, modification times, groups, owners,
device files and special files. -S means handle
sparse files efficiently. -H means preserve hard
links. -z means compress data for transfer; you
might want to omit this if your CPU is slow.
--numeric-ids is important as backups are
stored with the same numeric user and group IDs as the
original system; no translation via name is performed.
rsbackup(5) configuration man page
Configuration and cronjob structure for rsbackup.deb
Please report bugs via Github.
"
# this is kind of painful using only BREs
nroff -Tascii -man "$page" | ${GNUSED} \
'1d;$d;
1,/./{/^$/d};
s/&/\&/g;
s//\>/g;
s/@/\@/g;
s!\(.\)\1!\1!g;
s!\(&[#0-9a-z][0-9a-z]*;\)\1!\1!g;
s!_\(.\)!\1!g;
s!_\(&[#0-9a-z][0-9a-z]*;\)!\1!g;
s!<\1>!!g'
echo ""
if $stdhead; then
echo "@credits"
fi
echo " "
echo ""
mv $output.new $output
done
rsbackup-10.0/scripts/systest 0000775 0000000 0000000 00000011063 14407304317 0016403 0 ustar 00root root 0000000 0000000 #! /bin/sh
#
# Copyright © 2012, 2014, 2017 Richard Kettlewell.
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see