Backing Up Multiple Servers with Rsnapshot

Rsnapshot is a great little utility for creating easy-to-use backups. It creates a full-blown snapshot for each run, but uses hard links for files that haven’t changed. That means you get efficient, lightweight, incremental backups that look like exactly like full-blown copies (in every way).

This makes Rsnapshot a great choice for server backups. You can export your snapshots as read-only to the network, using NFS or Samba, and then your users can have full access to the backups, all of the time. They can see the file as it existed yesterday, last week, or last month, using only their native file manager. My clients have been very happy with this setup. Contrast that with BackupPC, another great Open Source backup product based on rsync. BackupPC uses special index files and tarballs, so you can’t just navigate the hierarchy at a command-line or file manager, and it requires a password to access a special web-based GUI.

As a sysadmin, I like the fact that Rsnapshot is basically a thin wrapper around rsync. All of the arguments, options, and features of rsync are inherited, and it feels very familiar. It can be easily installed with:

# liblchown-perl is needed, even though rsnapshot doesn't list it as required.
sudo apt-get install rsnapshot liblchown-perl

 
Risky Defaults

Unfortunately, there are a couple of problems with the default configuration that comes with Rsnapshot. But luckily, they are easy to fix.

First, there is an option called sync_first, which is off by default. This option must be on, under all circumstances. It shouldn’t really even be an option. If it’s off (like the default), then your incrementals are always rotated forward, even if the backup run fails. So, if you get an automated backup running, and then for some reason it breaks (but the cron job keeps running), you’ll will eventually lose all snapshots. It would not be fun to explain to clients that there was no backup. “But it was working fine last month, I don’t know where all the backups went…” By setting sync_first to true, you will ensure that the backups are only rotated if the backup run was successful (and we were able to grab a full snapshot).

Second, there is a problem in the way Rsnapshot handles multiple hosts. The config file allows you to specify multiple hosts and backup configurations, however, since Rsnapshot just runs via cron, all your hosts are forced to share the same backup schedule. By default, you can’t have one host that runs every hour, and another host that runs every 24 hours. Furthermore, if you are backing up a host with a slow network connection (or large data size), then all other hosts are blocked until that slow one is finished. By default, there is no way to run multiple hosts concurrently. Fortunately, these problems are easy to fix.

Finally, the example config for Rsnapshot does not show you how to do a secure SSH deployment, for network encryption. The example shown here shows how to set up your SSH keys in a way that limits the access of the Rsnaphost server.

Setting up an Enhanced Deployment

The trick around these default limitations is to use multiple, independent instances of Rsnapshot; basically, a unique config file for each host that we want to back up. We use a global Rsnapshot config in /etc/ to set common, global default values (like sync_first and any SSH options). Then, for each host we simply import those globals and then adjust a few host-specific options in its own config file.

The first step is to edit the default, global /etc/rsnapshot.conf file so that it’s just a common include file:

#
# On the backup host where Rsnapshot will run:
#
cd /etc

# Make a backup of the default, for comparisons:
cp -af rsnapshot.conf rsnapshot.conf-dist

# Create the common conf file:
mv rsnapshot.conf rsnapshot-common.conf

Next, edit the new rsnapshot-common.conf file with the changes below. These fix a couple of gotchas, and remove the global host configuration options. Note: THIS FILE MUST ONLY HAVE TABS. NO SPACES. (It’s a quirk of Rsnapshot that it requires tabs.)

-- rsnapshot.conf-dist 2010-05-11 23:50:09.000000000 -0700++ rsnapshot-common.conf 2011-06-20 16:54:53.799014415 -0700
 
    [ Starting on line 41... ]    [ Starting on line 41... ]
41#41#
42# See the README file or the man page for more details.42# See the README file or the man page for more details.
43#43#
44#cmd_cp /bin/cp 44cmd_cp /bin/cp
4545
46# uncomment this to use the rm program instead of the built-in perl routine.46# uncomment this to use the rm program instead of the built-in perl routine.
47#47#
 
    [ Starting on line 55... ]    [ Starting on line 55... ]
55# Uncomment this to enable remote ssh backups over rsync.55# Uncomment this to enable remote ssh backups over rsync.
56#56#
57#cmd_ssh /usr/bin/ssh57#cmd_ssh /usr/bin/ssh
 58cmd_ssh /usr/bin/ssh
5859
59# Comment this out to disable syslog support.60# Comment this out to disable syslog support.
60#61#
 
    [ Starting on line 86... ]    [ Starting on line 87... ]
86# i.e. hourly, daily, weekly, etc. #87# i.e. hourly, daily, weekly, etc. #
87#########################################88#########################################
8889
89interval hourly 6 90###interval hourly 6
90interval daily 7 91###interval daily 7
91interval weekly 4 92###interval weekly 4
92#interval monthly 3 93###interval monthly 3
9394
94############################################95############################################
95# GLOBAL OPTIONS #96# GLOBAL OPTIONS #
 
    [ Starting on line 103... ]    [ Starting on line 104... ]
103# 4 Extra Verbose Show extra verbose information104# 4 Extra Verbose Show extra verbose information
104# 5 Debug mode Everything105# 5 Debug mode Everything
105#106#
106verbose 2 107verbose 3
107108
108# Same as "verbose" above, but controls the amount of data sent to the109# Same as "verbose" above, but controls the amount of data sent to the
109# logfile, if one is being used. The default is 3.110# logfile, if one is being used. The default is 3.
 
    [ Starting on line 120... ]    [ Starting on line 121... ]
120# If you enable this, make sure the lockfile directory is not world121# If you enable this, make sure the lockfile directory is not world
121# writable. Otherwise anyone can prevent the program from running.122# writable. Otherwise anyone can prevent the program from running.
122#123#
123lockfile /var/run/rsnapshot.pid 124###lockfile /var/run/rsnapshot.pid
124125
125# Default rsync args. All rsync commands have at least these options set.126# Default rsync args. All rsync commands have at least these options set.
126#127#
 
    [ Starting on line 174... ]    [ Starting on line 175... ]
174# details. The default is 0 (off).175# details. The default is 0 (off).
175#176#
176#sync_first 0177#sync_first 0
 178sync_first 1
177179
178# If enabled, rsnapshot will move the oldest directory for each interval180# If enabled, rsnapshot will move the oldest directory for each interval
179# to [interval_name].delete, then it will remove the lockfile and delete181# to [interval_name].delete, then it will remove the lockfile and delete
 
    [ Starting on line 193... ]    [ Starting on line 195... ]
193###############################195###############################
194196
195# LOCALHOST197# LOCALHOST
196backup /home/ localhost/ 198###backup /home/ localhost/
197backup /etc/ localhost/ 199###backup /etc/ localhost/
198backup /usr/local/ localhost/ 200###backup /usr/local/ localhost/
199#backup /var/log/rsnapshot localhost/201#backup /var/log/rsnapshot localhost/
200#backup /etc/passwd localhost/202#backup /etc/passwd localhost/
201#backup /home/foo/My Documents/ localhost/203#backup /home/foo/My Documents/ localhost/

 

Notice that the common config file above does not specify any actual backup hosts, because each host now gets its own rsnapshot.conf file. In the example below, we make a special directory /home/rsnapshot/ as the root for all backup hosts, but you can use whatever directory you like.

Here are the commands to run for each host:

# 
# For each new host to back up, run these as root:
# 

#
# First, specify the hostname (used in the config files).  
# This hostname must actually work.
HOST="cst6"

#
# Make the directory to hold the backups.
# Notice that in this example, everything is under /home/rsnapshot/:
mkdir -p /home/rsnapshot/$HOST
mkdir -p /home/rsnapshot/$HOST/.sync

# 
# Here we create the unique Rsnapshot config file for the new host.
# The first line includes the global defaults in /etc/rsnapshot-common.conf.
#
# You may want to tweak these settings, as by default it runs every hour
# and retains incrementals for up to ten years.
#
# Also notice that each host gets its own .log and .pid file, 
# and also an "exclude_file.txt" that lists which files to ignore.
#
# (Note: The -e option to echo makes \t work as a tab)
echo -e "
include_conf\t/etc/rsnapshot-common.conf

snapshot_root\t/home/rsnapshot/$HOST/

interval\thourly\t24
interval\tdaily\t7
interval\tweekly\t4
interval\tmonthly\t120

logfile\t/home/rsnapshot/$HOST/rsnapshot.log
lockfile\t/home/rsnapshot/$HOST/rsnapshot.pid

exclude_file\t/home/rsnapshot/$HOST/exclude_file.txt

backup\troot@$HOST:/\t./
" >> /home/rsnapshot/$HOST/rsnapshot.conf

#
# Here we create the "exclude" file for this host.
# This is in the rsync format for exclude files, which is a
# little tricky and not easy to understand.
# 
echo "- /sys
- /proc
- /tmp
- /dev
- /var/lib/libvirt/images
- /var/lock
- /var/run
- /home/rsnapshot
+ /*
" >> /home/rsnapshot/$HOST/exclude_file.txt

# 
# Next, we add this host to root's crontab file.
# In this example, we keep a working copy of root's crontab in
# /root/crontab.root.
# 
# Note that this is not the only way to set up cron.  Another
# example would be to use cron's run-parts to manage the hourly,
# daily, and weekly runs separately.  My preference is to keep each
# backup host's cron entries all together in one file, as in this example,
# but it is not required.  (Thanks to Nico Kadel-Garcia for pointing this out.)
# 
# You'll probably want to edit this for hourly or daily:
# 
echo "
# m h  dom mon dow   command
00 * * * *  rsnapshot -c /home/rsnapshot/$HOST/rsnapshot.conf sync && rsnapshot -c /home/rsnapshot/$HOST/rsnapshot.conf hourly
00 04 * * * rsnapshot -c /home/rsnapshot/$HOST/rsnapshot.conf daily
00 02 * * 0 rsnapshot -c /home/rsnapshot/$HOST/rsnapshot.conf weekly
00 00 1 * * rsnapshot -c /home/rsnapshot/$HOST/rsnapshot.conf monthly
" >> /root/crontab.root

# 
# Finally, we copy the Rsnapshot root's public key to the new
# host we are going to back up. This is so it can be trusted by the host
# to run backups.
cd /root
# 
# If this is your first time setting up a host, you may need
# to create a key with this command:
### ssh-keygen -t rsa
#
scp .ssh/id_rsa.pub ubuntu@$HOST:root-$HOSTNAME.pub

 
Now we need to tell the new target host to trust the Rsnapshot SSH key.

Security here is critical. We want the new host to trust the Rsnapshot key, as root. It needs to be trusted by root so that rsync can grab all the files, even the ones owned by other users, with different UIDs and GIDs.

But if our backup server (the Rsnapshot server) is compromised, we do not want the attacker to have an SSH key that grants root access to all of our primary hosts. So we use a couple of SSH options to restrict access.

The first option is in /etc/ssh/sshd_config:

#
# In /etc/ssh/sshd_config, on every new host you back up:
# 
PermitRootLogin forced-commands-only

This option tells SSH that root cannot log in with a regular shell, even when using a trusted key. Instead, a particular trusted key can only run a forced command — in this case, the rsync –server command, in read-only mode.

This way, if our backup server is compromised with a local root shell, the attacker will (sadly) have access to all our backups, and he can initiate a new backup run, but that is all he can do. He is not able to log in to the primary hosts, and he’s not able to change any of the files on the primary host. If you’re lucky, that will buy you enough time to detect the intrusion and shut down the compromised system.

#
# Now trust the Rsnapshot key, but only to do backups.
# Run this on every new host you back up:
# 
mkdir -p /root/.ssh
chmod 700 /root/.ssh


#
# This little perl script restricts access to read-only rsync.
# It comes as part of the rsync documentation, so we unzip it
# into /root/.
#
zcat /usr/share/doc/rsync/scripts/rrsync.gz > /root/rrsync
chmod 700 /root/rrsync

#
# Now we trust the new key. The options to SSH are:
#  no-pty: Don't offer the user a psuedo terminal.  This adds a layer against shell logins.
#  no-agent-forwarding: Don't let an attacker use their own agent.  (Doesn't really add much here.)
#  no-X11-forwarding: Prevents X11 apps.  Just another extra layer to reduce attack vectors.
#  no-port-forwarding: Do not allow SSH tunnels using this key.
#  command="/root/rrsync -ro /" : This forces the script to run on login, with the -ro "read-only" option
# 
echo -n 'no-pty,no-agent-forwarding,no-X11-forwarding,no-port-forwarding,command="/root/rrsync -ro /" ' >> /root/.ssh/authorized_keys
cat /home/ubuntu/root-BACKUPSERVER.pub >> /root/.ssh/authorized_keys
# SSH has very strict permission requirements:
chmod 600 /root/.ssh/authorized_keys

 
Now confirm that it looks right in authorized_keys:
root@cst6:~# cat /root/.ssh/authorized_keys
command="/root/rrsync -ro /" ssh-rsa AAAAB3[...snip...]raIw== root@www-backup

Now trust is almost established. You just need to tell the Rsnapshot host to trust the host key of the new server. This is done by attempting to SSH there as root:

# Run this on the Rsnapshot host to accept the new host key:
ssh root@$HOST

# You should be prompted to accept the host's key, by typing "yes".
# You want to get as far as having the PTY denied to you:
#root@cst8:/root# ssh cst6
#PTY allocation request failed on channel 0

Finally, launch the first backup run manually. This is to make sure everything goes smoothly, and so you can see how long a full backup takes. If everything goes well, you can install the updated crontab:

# Back on the rsnapshot server:
# Start a backup

# Always start long-lived operations in a screen session.
screen

# Tell the new screen session this var
HOST="cst6"
cd /home/rsnapshot/$HOST/
echo "START: "`date` >> first_sync.log ; cat first_sync.log ; time rsnapshot -c rsnapshot.conf -vv sync ; echo "END: "`date` >> first_sync.log ; echo "DONE at "`date`

# Finally, if everything went without error, activate cron with the new schedule:
crontab /root/crontab.root

Comments are closed.