Interservice communication with Redis and Sidekiq remote workers

By Harlow Ward

As our infrastructure continues to grow (and we continue to extract core services) we’re consistently having to sync data between applications.

One of the most common mistakes we see when a team is initially structuring their application as an SOA is to lean on a single database as the central storage bucket backing all of the services.
- Tammer Saleh (SOA AntiPatterns Talk)

A message queue allows us to decouple the services, and acts as a great temporary holding place while transferring data between applications.

The Gateway Service is a core component of our infrastructure as its responsible for interacting with all of our external hotel partners and making sure we always have the most up to date inventory.

message queue implementation

When a message arrives at the Rails App there is a small (yet not insignificant) amount of work that needs to be done before persisting it to the data-store.

A scheduled job was used to batch import messages into Sidekiq for parallel asynchronous processing. However, as the volume of messages grew, the scheduled job became a bottleneck in our pipeline.

message queue volume

We decided to explore the idea of pushing jobs directly from the Gateway Service directly into the Rails App Sidekiq queue.

remote worker implementation

A thin layer was created to manage the Redis connection and wrap the interface of the Sidekiq client.

class RemoteWorker
  cattr_writer :client

  def self.client
    @@client or raise "Please set the client to a Sidekiq::Client connection"
  end

  def self.push(worker_name, attrs = {}, queue_name = "default")
    client.push(args: [attrs], class: worker_name, queue: queue_name)
  end
end

Some feedback on the PR reminded me we need to namespace the Sidekiq queue.

use namespace for sidekiq queues

Namespacing allowed us to run multiple workers simultaneously in our local environment – This gave us the ability to test an end-to-end integration in development mode.

# config/initializers/remote_worker.rb
url = ENV.fetch("REMOTE_WORKER_REDIS_URL")
namespace = ENV.fetch("REMOTE_WORKER_REDIS_NAMESPACE")
redis = Redis::Namespace.new(namespace, redis: Redis.new(url: url))

RemoteWorker.client = Sidekiq::Client.new(ConnectionPool.new{ redis })

It’s worth noting that we’ve now introduced connaissance of name between the two services – The application pushing remote work needs to know the class name of the worker in the remote application.

To help avoid leaking this knowledge throughout the codebase we created objects to wrap the internals of each remote worker, and added validations to help enforce data integrity.

class HotelRatesRemoteWorker
  include ActiveModel::Model
  validates :hotel_id, numericality: { only_integer: true }
  validates :source_name, presence: true
  # ...

  def push
    if valid?
      RemoteWorker.push(worker_name, worker_args)
    else
      raise ActiveModel::StrictValidationFailed, errors.full_messages
    end
  end

  private

  def worker_name
    "HotelRatesWorker"
  end

  def worker_args
    {
      hotel_id: hotel_id,
      source_name: source_name,
      # ...
    }
  end
end

The remote worker setup has allowed us to remove the duplicative message queue, and by doing this we’ve increased the throughput by continually streaming jobs directly into the remote Sidekiq worker queues.

Written by Harlow Ward

Read more posts by Harlow, and follow Harlow on Twitter.

Interested in building something great?

Join us in building the worlds most loved hotel app.
View our open engineering positions.