Resuming File Transfers in RSync

RSync is a great tool for archiving and backup. But how do you get it to cope with connection problems? For example, you are trying to backup a large archive file but you have a poor connection which keeps dropping and so your transfer has to start all over again. Well here’s what I use:

rsync -avz --progress --partial -e ssh /home/mybackup/* <email address>@<rsync host>:<remote folder name>

Note that the email address does not require quote marks around it. Notice I do not use –append or –append-verify. If I did it would assume that the start of the file has remained the same (which might be the case for a log file or similar). This isn’t the case for me. I also don’t use the –delete option because this would simply delete the temporary file left by –partial.

It works a treat but I would recommend a break of 15 minutes when attempting to resume the transfer (the time will vary depending upon how the remote server deals with temporary files).

For more on this and a very useful bash script, have a look here:
http://superuser.com/questions/302842/resume-rsync…

Rails 5 + Puma + Nginx + ActionCable

In development mode things seemed to take care of themselves. Production mode was a different matter. Let’s just say it was a sensitive and delicate configuration. So in documenting this I’ve tried to condense things to the bare minimum and using the simplest configuration: hosting the “cable” within the host application (i.e. in the same virtual host). For full ActionCable documentation please visit: Rails Guides

Make sure you have a cable.js file (app/assets/javascripts/cable.js)

The “rails new” command should have created this for you.

Create a handler (app/assets/javascripts/channels/chat_channel.coffee)

App.chat = App.cable.subscriptions.create "ChatChannel",
  connected: ->
    alert 'connected'

  disconnected: ->
    alert 'disconnected'

  received: (data) ->
    # do you screen update stuff here

I’ve included popup alerts. These are quite useful when you’re doing this for the first time so you know your basic connection is working.

Make sure you have your channel and connection ActionCable files

Check for app/channels/application_cable/channel.rb and app/channels/application_cable/connection.rb. These should have been created for you. If you’re running ActionCable within your application (rather than on a separate server/vhost) you can add authentication for the connection thus:

module ApplicationCable
  class Connection < ActionCable::Connection::Base
    identified_by :current_user

    def connect
      self.current_user = find_verified_user
    end

    protected
    def find_verified_user
      if current_user = User.find_by(id: cookies.signed[:user_id])
        current_user
      else
        reject_unauthorized_connection
      end
    end
  end
end

Create your own channel file (app/channels/chat_channel.rb)

Here’s where you subscribe to your own broadcast.

class ChatChannel < ApplicationCable::Channel
  def subscribed
    stream_from 'chat_channel'
  end

  def unsubscribed
    # Any cleanup needed when channel is unsubscribed
  end
end

Set your configuration parameters

There are 2 files to change. First we must update config/environments/production.rb and set the config.action_cable constants:

config.action_cable.url = 'ws://myapp.spannersoftware.com/cable' # ws:// is non-secure, wss:// is secure
config.action_cable.allowed_request_origins = [ 'http://myapp.spannersoftware.com' ]

Important: do not add a trailing slash to either of these!

Next update config/cable.yml and set the host and port for the redis server:

development:
  adapter: async

test:
  adapter: async

production:
  adapter: redis
  url: redis://localhost:6379/1

You may not need to change this.

Add the ActionCable metatag

In app/views/layouts/application.html.erb add the following line in the <head> section:

<%= action_cable_meta_tag %>

Install and fire up the Redis server

Redis is a cool bit of kit for providing efficient messaging, for more information go here: Redis.io.

apt-get install redis-server
service redis-server start

Edit /etc/redis/redis.conf to set the port to 6379 and bind to 127.0.0.1.

Configure Nginx and Puma

Here is my Nginx virtual host example:

upstream my_app {
 server unix:/home/myapp/tmp/myapp.sock;
}
server {
 listen 10.10.10.10:80;
 server_name myapp.spannersoftware.com;

 access_log /home/myapp/log/access.log;
 error_log /home/myapp/log/error.log;
 root /home/myapp/public;

 location / {
 try_files /maint.html $uri @ruby;
 }

 location @ruby {
 proxy_pass http://my_app;
 proxy_set_header X-Real-IP $remote_addr;
 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
 proxy_set_header Host $http_host;
 proxy_redirect off;
 }

 location /cable {
 proxy_pass http://my_app;
 proxy_http_version 1.1;
 proxy_set_header Upgrade $http_upgrade;
 proxy_set_header Connection "upgrade";
 }
}

Here is my Puma config (config/puma.rb):

threads_count = ENV.fetch("RAILS_MAX_THREADS") { 5 }.to_i
threads threads_count, threads_count
bind "unix:/home/myapp/tmp/bluebird2.sock"
environment ENV.fetch("RAILS_ENV") { "production" }
workers ENV.fetch("WEB_CONCURRENCY") { 2 }
daemonize true
pidfile '/home/myapp/tmp/pids/puma.pid'
I'm using a Unix socket here, the default environment is production, the process is daemonized and I've specified a pidfile location. The rest of the settings are default.

Set your app running

Assuming you have the puma gem installed correctly you should be able to run your application complete with working ActionCable.

cd /home/myapp
puma -C config/puma.rb -e production

You should see the message “connected” pop up in your browser when you first load the page. Hopefully you’ve found this helpful but please let me know if I’ve missed anything.

 

Nested Attributes Not Setting Child IDs

Developing in Rails 5 recently I was struggling to think of why my child record IDs were not being set automatically on create. I thought this happened automatically. Apparently it does sometimes but not every time. Puzzling!

To fix the problem I had to use inverse_of. Here’s how:

class Publisher < ApplicationRecord
  has_many :articles, index_errors: true, dependent: :destroy, inverse_of: :publisher
  accepts_nested_attributes_for :articles, allow_destroy: true

class Article < ApplicationRecord
  belongs_to :publisher, inverse_of: :articles
  has_many :comments, index_errors: true, dependent: :destroy, inverse_of: :article
  accepts_nested_attributes_for :comments, allow_destroy: true, reject_if: proc { |item| item[:message].blank? }

class Comment < ApplicationRecord
  belongs_to :article, inverse_of: :comments

It seems strange to have to do this (and no doubt it will be fixed in future) but without it my application complained of missing parent IDs: publisher_id (for Article records) and article_id (for Comment records).