Resuming File Transfers in RSync

RSync is a great tool for archiving and backup. But how do you get it to cope with connection problems? For example, you are trying to backup a large archive file but you have a poor connection which keeps dropping and so your transfer has to start all over again. Well here’s what I use:

rsync -avz --progress --partial -e ssh /home/mybackup/* <email address>@<rsync host>:<remote folder name>

Note that the email address does not require quote marks around it. Notice I do not use –append or –append-verify. If I did it would assume that the start of the file has remained the same (which might be the case for a log file or similar). This isn’t the case for me. I also don’t use the –delete option because this would simply delete the temporary file left by –partial.

It works a treat but I would recommend a break of 15 minutes when attempting to resume the transfer (the time will vary depending upon how the remote server deals with temporary files).

For more on this and a very useful bash script, have a look here:
http://superuser.com/questions/302842/resume-rsync…

Automated Rsync over SSH

The Mission

Let’s say we want to set-up rsync over SSH to securely backup a folder from one server to another. Our aim¬†is to run a daily backup in cron with a command like this:

rsync -avz -e ssh /home/somefolder/important_files remoteuser@target_host:/home/mybackup/important_files_copy

On the Target Machine

The following steps work well for me but change as you wish (you may wish to create a folder under an existing user for example):

  1. Create a folder for the backup user:
    cd /home
    mkdir mybackupuser
  2. Next create the user, update the folder permissions and set the user password (make a note of this password):
    useradd -d /home/mybackupuser mybackupuser
    chown mybackupuser.root mybackupuser
    chmod 750 mybackupuser
    passwd mybackupuser
  3. Update the sshd_config file to allow this user to be used for ssh logins:
    vi /etc/ssh/sshd_config

    Add or amend the “AllowUsers” directive:

    AllowUsers mybackupuser
  4. Create the hidden folder for authorized keys:
    cd /home/mybackupuser
    mkdir .ssh
    chown mybackupuser.root .ssh
    chmod 700 .ssh
  5. Create the security script (attempt to filter any ssh command except rsync –server):
    cd /home/mybackupuser
    vi valid-rsync

    Copy and paste the following lines and then save the file:

    #!/bin/sh
    
    case "$SSH_ORIGINAL_COMMAND" in
    *\&*)
    echo "Rejected"
    ;;
    *\(*)
    echo "Rejected"
    ;;
    *\{*)
    echo "Rejected"
    ;;
    *\;*)
    echo "Rejected"
    ;;
    *\<*)
    echo "Rejected"
    ;;
    *\`*)
    echo "Rejected"
    ;;
    *\|*)
    echo "Rejected"
    ;;
    rsync\ --server*)
    $SSH_ORIGINAL_COMMAND
    ;;
    *)
    echo "Rejected"
    ;;
    esac

    I cannot claim credit for writing the above script but unfortunately I can’t remember where I got it from originally. Suffice to say it seems to be quite widely distributed on the web. To whoever wrote it: thank you!

  6. Set privileges for this file:
    chown mybackupuser.root valid-rsync
    chmod 700 valid-rsync

On the Source Machine (mostly!)

Now, in case you were getting bored, we move to the source machine to create an ssh key. The public part of this key will then be copied to the target machine.

  1. Log in as the user that your backups will run under. For example, if your backup script will run under the root user (usual) log in as root (or sudo su).
  2. Generate an SSH key pair:
    ssh-keygen -t rsa -b 2048

    Or, if it’s not your own server then you might want to specify a path for the key generation (rather than the default which is ~/.ssh/id_rsa):

    ssh-keygen -t rsa -b 2048 -f /home/someuser/rsync-key
  3. Important: accept all defaults and just press [Enter] when asked for the passphrase
  4. Upload the file from the source machine to the target machine using secure copy:
    scp ~/.ssh/id_rsa.pub mybackupuser@target_host:.ssh/authorized_keys

    If you specified a path (the -f option) then use the following command:

    scp /home/someuser/rsync-key.pub mybackupuser@target_host:/home/mybackupuser

    Now append the contents of rsync-key.pub to /home/mybackupuser/.ssh/authorized_keys on the target machine.

  5. On the target machine, prefix the key with additional security measures including the ip address and valid-rsync command file that you created earlier:
    from="xx.xx.xx.xx",command="/home/jjbb/valid-rsync" ssh-rsa ASHAKJSDHhaisudhfaksjfhHAISUDHiauegfkjaHSDKJHDjkh.....

    “xx.xx.xx.xx” is the IP address of the source machine.

  6. Now you are ready to rsync:
    rsync -avz -e ssh /home/somefolder/important_files remoteuser@target_host:/home/mybackup/important_files_copy

    If you are using your own file (see -f option earlier) you must add “-i /home/someuser/rsync-key” to the ssh options thus:

    rsync -avz -e "ssh -i /home/someuser/rsync-key" /home/somefolder/important_files remoteuser@target_host:/home/mybackup/important_files_copy