Resuming File Transfers in RSync

RSync is a great tool for archiving and backup. But how do you get it to cope with connection problems? For example, you are trying to backup a large archive file but you have a poor connection which keeps dropping and so your transfer has to start all over again. Well here’s what I use:

rsync -avz --progress --partial -e ssh /home/mybackup/* <email address>@<rsync host>:<remote folder name>

Note that the email address does not require quote marks around it. Notice I do not use –append or –append-verify. If I did it would assume that the start of the file has remained the same (which might be the case for a log file or similar). This isn’t the case for me. I also don’t use the –delete option because this would simply delete the temporary file left by –partial.

It works a treat but I would recommend a break of 15 minutes when attempting to resume the transfer (the time will vary depending upon how the remote server deals with temporary files).

For more on this and a very useful bash script, have a look here:
http://superuser.com/questions/302842/resume-rsync…

Rails 5 + Puma + Nginx + ActionCable

In development mode things seemed to take care of themselves. Production mode was a different matter. Let’s just say it was a sensitive and delicate configuration. So in documenting this I’ve tried to condense things to the bare minimum and using the simplest configuration: hosting the “cable” within the host application (i.e. in the same virtual host). For full ActionCable documentation please visit: Rails Guides

Make sure you have a cable.js file (app/assets/javascripts/cable.js)

The “rails new” command should have created this for you.

Create a handler (app/assets/javascripts/channels/chat_channel.coffee)

App.chat = App.cable.subscriptions.create "ChatChannel",
  connected: ->
    alert 'connected'

  disconnected: ->
    alert 'disconnected'

  received: (data) ->
    # do you screen update stuff here

I’ve included popup alerts. These are quite useful when you’re doing this for the first time so you know your basic connection is working.

Make sure you have your channel and connection ActionCable files

Check for app/channels/application_cable/channel.rb and app/channels/application_cable/connection.rb. These should have been created for you. If you’re running ActionCable within your application (rather than on a separate server/vhost) you can add authentication for the connection thus:

module ApplicationCable
  class Connection < ActionCable::Connection::Base
    identified_by :current_user

    def connect
      self.current_user = find_verified_user
    end

    protected
    def find_verified_user
      if current_user = User.find_by(id: cookies.signed[:user_id])
        current_user
      else
        reject_unauthorized_connection
      end
    end
  end
end

Create your own channel file (app/channels/chat_channel.rb)

Here’s where you subscribe to your own broadcast.

class ChatChannel < ApplicationCable::Channel
  def subscribed
    stream_from 'chat_channel'
  end

  def unsubscribed
    # Any cleanup needed when channel is unsubscribed
  end
end

Set your configuration parameters

There are 2 files to change. First we must update config/environments/production.rb and set the config.action_cable constants:

config.action_cable.url = 'ws://myapp.spannersoftware.com/cable' # ws:// is non-secure, wss:// is secure
config.action_cable.allowed_request_origins = [ 'http://myapp.spannersoftware.com' ]

Important: do not add a trailing slash to either of these!

Next update config/cable.yml and set the host and port for the redis server:

development:
  adapter: async

test:
  adapter: async

production:
  adapter: redis
  url: redis://localhost:6379/1

You may not need to change this.

Add the ActionCable metatag

In app/views/layouts/application.html.erb add the following line in the <head> section:

<%= action_cable_meta_tag %>

Install and fire up the Redis server

Redis is a cool bit of kit for providing efficient messaging, for more information go here: Redis.io.

apt-get install redis-server
service redis-server start

Edit /etc/redis/redis.conf to set the port to 6379 and bind to 127.0.0.1.

Configure Nginx and Puma

Here is my Nginx virtual host example:

upstream my_app {
 server unix:/home/myapp/tmp/myapp.sock;
}
server {
 listen 10.10.10.10:80;
 server_name myapp.spannersoftware.com;

 access_log /home/myapp/log/access.log;
 error_log /home/myapp/log/error.log;
 root /home/myapp/public;

 location / {
 try_files /maint.html $uri @ruby;
 }

 location @ruby {
 proxy_pass http://my_app;
 proxy_set_header X-Real-IP $remote_addr;
 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
 proxy_set_header Host $http_host;
 proxy_redirect off;
 }

 location /cable {
 proxy_pass http://my_app;
 proxy_http_version 1.1;
 proxy_set_header Upgrade $http_upgrade;
 proxy_set_header Connection "upgrade";
 }
}

Here is my Puma config (config/puma.rb):

threads_count = ENV.fetch("RAILS_MAX_THREADS") { 5 }.to_i
threads threads_count, threads_count
bind "unix:/home/myapp/tmp/bluebird2.sock"
environment ENV.fetch("RAILS_ENV") { "production" }
workers ENV.fetch("WEB_CONCURRENCY") { 2 }
daemonize true
pidfile '/home/myapp/tmp/pids/puma.pid'
I'm using a Unix socket here, the default environment is production, the process is daemonized and I've specified a pidfile location. The rest of the settings are default.

Set your app running

Assuming you have the puma gem installed correctly you should be able to run your application complete with working ActionCable.

cd /home/myapp
puma -C config/puma.rb -e production

You should see the message “connected” pop up in your browser when you first load the page. Hopefully you’ve found this helpful but please let me know if I’ve missed anything.

 

Automated Rsync over SSH

The Mission

Let’s say we want to set-up rsync over SSH to securely backup a folder from one server to another. Our aim is to run a daily backup in cron with a command like this:

rsync -avz -e ssh /home/somefolder/important_files remoteuser@target_host:/home/mybackup/important_files_copy

On the Target Machine

The following steps work well for me but change as you wish (you may wish to create a folder under an existing user for example):

  1. Create a folder for the backup user:
    cd /home
    mkdir mybackupuser
  2. Next create the user, update the folder permissions and set the user password (make a note of this password):
    useradd -d /home/mybackupuser mybackupuser
    chown mybackupuser.root mybackupuser
    chmod 750 mybackupuser
    passwd mybackupuser
  3. Update the sshd_config file to allow this user to be used for ssh logins:
    vi /etc/ssh/sshd_config

    Add or amend the “AllowUsers” directive:

    AllowUsers mybackupuser
  4. Create the hidden folder for authorized keys:
    cd /home/mybackupuser
    mkdir .ssh
    chown mybackupuser.root .ssh
    chmod 700 .ssh
  5. Create the security script (attempt to filter any ssh command except rsync –server):
    cd /home/mybackupuser
    vi valid-rsync

    Copy and paste the following lines and then save the file:

    #!/bin/sh
    
    case "$SSH_ORIGINAL_COMMAND" in
    *\&*)
    echo "Rejected"
    ;;
    *\(*)
    echo "Rejected"
    ;;
    *\{*)
    echo "Rejected"
    ;;
    *\;*)
    echo "Rejected"
    ;;
    *\<*)
    echo "Rejected"
    ;;
    *\`*)
    echo "Rejected"
    ;;
    *\|*)
    echo "Rejected"
    ;;
    rsync\ --server*)
    $SSH_ORIGINAL_COMMAND
    ;;
    *)
    echo "Rejected"
    ;;
    esac

    I cannot claim credit for writing the above script but unfortunately I can’t remember where I got it from originally. Suffice to say it seems to be quite widely distributed on the web. To whoever wrote it: thank you!

  6. Set privileges for this file:
    chown mybackupuser.root valid-rsync
    chmod 700 valid-rsync

On the Source Machine (mostly!)

Now, in case you were getting bored, we move to the source machine to create an ssh key. The public part of this key will then be copied to the target machine.

  1. Log in as the user that your backups will run under. For example, if your backup script will run under the root user (usual) log in as root (or sudo su).
  2. Generate an SSH key pair:
    ssh-keygen -t rsa -b 2048

    Or, if it’s not your own server then you might want to specify a path for the key generation (rather than the default which is ~/.ssh/id_rsa):

    ssh-keygen -t rsa -b 2048 -f /home/someuser/rsync-key
  3. Important: accept all defaults and just press [Enter] when asked for the passphrase
  4. Upload the file from the source machine to the target machine using secure copy:
    scp ~/.ssh/id_rsa.pub mybackupuser@target_host:.ssh/authorized_keys

    If you specified a path (the -f option) then use the following command:

    scp /home/someuser/rsync-key.pub mybackupuser@target_host:/home/mybackupuser

    Now append the contents of rsync-key.pub to /home/mybackupuser/.ssh/authorized_keys on the target machine.

  5. On the target machine, prefix the key with additional security measures including the ip address and valid-rsync command file that you created earlier:
    from="xx.xx.xx.xx",command="/home/jjbb/valid-rsync" ssh-rsa ASHAKJSDHhaisudhfaksjfhHAISUDHiauegfkjaHSDKJHDjkh.....

    “xx.xx.xx.xx” is the IP address of the source machine.

  6. Now you are ready to rsync:
    rsync -avz -e ssh /home/somefolder/important_files remoteuser@target_host:/home/mybackup/important_files_copy

    If you are using your own file (see -f option earlier) you must add “-i /home/someuser/rsync-key” to the ssh options thus:

    rsync -avz -e "ssh -i /home/someuser/rsync-key" /home/somefolder/important_files remoteuser@target_host:/home/mybackup/important_files_copy