Rsync is a fast and extraordinarily versatile file copying tool. It can copy locally, to/from another host over any remote shell, or to/from a remote rsync daemon.
It offers a large number of options that control every aspect of its behavior and permit very flexible specification of the set of files to be copied. It is famous for its delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination. Rsync is widely used for backups and mirroring and as an improved copy command for everyday use.
Rsync finds files that need to be transferred using a "quick check" algorithm (by default) that looks for files that have changed in size or in last-modified time.
Any changes in the other preserved attributes (as requested by options) are made on the destination file directly when the quick check indicates that the file's data does not need to be updated.
- rsync command examples: http://www.thegeekstuff.com/2010/09/rsync-command-examples/
I will show shortly how to:
- backup your files from remote to local using rsync
- use modules to have multiple backups possible
- secure the connection with ssh
- avoid to prompt for the password, so that your backup can be done via script/cronjob
Before we start, I'll call "local" the computer where the files have to be copied and "remote" the computer where those files are stored and where you have to listen for ssh connections.
To secure our data, we'll use rsync via a remote ssh connection, so there's no need to start rsync as a daemon, but sshd must be configured to accept connections without password and
rsa-key authentication must be enabled in your
PermitRootLogin without-password PubkeyAuthentication yes AllowUsers root
Here "root" is the only user who is allowed to connect via ssh. So the user "root" will be used at the ssh level and should not be confused with "rsync-user", which will be used to log-in to the
site1 in the following example.
Log-in as "root" and create the config file
# common stuff motd file = /etc/rsyncd_motd # the following in case you want to test rsync as daemon log file = /var/log/rsyncd.log pid file = /var/run/rsyncd.pid lock file = /var/run/rsyncd.lock [site1] # this is the path of the files to backup path = /home/ssh-user/path/where/site1/files/live comment = site1 files uid = root gid = root read only = yes list = yes auth users = rsync-user secrets file = /root/rsyncd.scrt # we don't have super user access use chroot = false [site2] [....site2 stuff....]
uid and gid are the userID and the groupID under which file transfers will take place.
Before the transfer will start, you have to authenticate rsync with "auth user". Create the secret file
~/rsync.scrt which holds the
Remove the 'r' flag to other users:
chmod o-r ~/rsync.scrt
Since we want to backup our files by means of a script and a cronjob, it's important that the remote
ssh connection will not prompt for any password. We can achieve this by exchanging a ssh-key between client and server.
Create the private and public keys:
root@localhost:~# ssh-keygen Generating public/private rsa key pair. Enter file in which to save the key (/root/.ssh/id_rsa_remoteHost): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /root/.ssh/id_rsa_remoteHost. Your public key has been saved in /root/.ssh/id_rsa_remoteHost.pub. The key fingerprint is: a0:53:33:c5:d1:ea:4c:e2:a1:98:d9:ba:b0:e8:5f:90 root@localhost The key's randomart image is: +--[ RSA 2048]----+ | o++o | | o. . | | . .. | | .oo. | | E.O .S | | * * | |. . o . | |.o. . . | |+.oo | +-----------------+
Now you have to append the public key
id_rsa_remoteHost.pub to the remote server's
ssh-copy-id is a program which can do this for you:
root@localhost:~# ssh-copy-id -i ~/.ssh/id_rsa_remoteHost -p 12345 root@remoteHost
Now test that the connection is allowed with no password:
root@localhost:~# ssh -p 12345 -l root -i /root/.ssh/id_rsa_remoteHost <remoteHost> Last login: Mon Sep 2 16:04:57 2013 from localhost Linux 22.214.171.124-vs126.96.36.199.29.2-smp. root@remotehost:~#
Now we are ready to create our backup shell-script as
#!/bin/sh /usr/bin/rsync \ -avz --exclude "*~" --delete-after \ -e "ssh -p 12345 -l root -i /root/.ssh/id_rsa_remoteHost" \ --password-file /root/remoteHost_rsync_pwd \ rsync-user1@::site1 \ /local/destination/path
Remember to give the flag +x to that file:
chmod +x /usr/local/bin/rsync_backup.sh
The password file /root/remoteHost_rsync_pwd holds the password of the rsync connection; in this way our shell-script will not receive a password prompt when it connects. It should be stored in a safe place and priviledges must be given only to the root user. It will contain just the password string.
Maybe the line
-avz --exclude "*~" \
deserves some description, but you are invited to refer to the man page for more details.
- --exclude "*~" is to avoid the copy of backup files of my text editor
- "-a" stands for -rlptgoD and preserves everything
- "-r" means recursive mode while traversing directories
- "-p", "-o", and "-g" preserve the permissions, owner and group information of files and directories to be copied
- "-t" preserves the file and directory timestamps
- "-l" preserves the symbolic links
- "-D" preserves devices and special files
- "-v" turns on verbosity in output
- "-z" enables compression
Connecting to the remote Host
You can have a quick connection to the remote Host if you setup a ~/.ssh/config file as follow
Host MyHost HostName remoteHost.net User ssh-user Port 12345 IdentityFile ~/.ssh/id_rsa_remoteHost
and connecting as
> ssh MyHost Enter passphrase for key '/home/ssh-user/.ssh/id_rsa_remoteHost': Last login: Mon Sep 2 16:04:57 2013 from localhost Linux 188.8.131.52-vs184.108.40.206.29.2-smp. ssh-user@remotehost:~#
At this point it is convenient to disable root remote access setting
/etc/ssh/sshd_config as follow:
PermitRootLogin without-password AllowUsers ssh-user PubkeyAuthentication yes