Ruby Concurrency and Parallelism: A Practical Primer

[this article was written by eqbal quran - software engineer @ toptal ]

let’s start by clearing up an all-too-common point of confusion; namely: concurrency and parallelism are not the same thing (i.e., concurrent != parallel).

in particular, concurrency is when two tasks can start, run, and complete in overlapping time periods. it doesn’t necessarily mean, though, that they’ll ever both be running at the same instant (e.g., multiple threads on a single-core machine). in contrast, parallelism is when two tasks literally run at the same time (e.g., multiple threads on a multicore processor).

the key point here is that concurrent processes and/or threads will not necessarily be running in parallel.

this post provides a practical (rather than theoretical) treatment of the various techniques and approaches that are available for concurrency and parallelism in ruby.

our test case

for a simple test case, i’ll create a mailer class and add a fibonacci function (rather than the sleep() method) to make each request more cpu-intensive, as follows:

class mailer

  def self.deliver(&block)
    mail = mailbuilder.new(&block).mail
    mail.send_mail
  end

  mail = struct.new(:from, :to, :subject, :body) do 
    def send_mail
      fib(30)
      puts "email from: #{from}"
      puts "email to  : #{to}"
      puts "subject   : #{subject}"
      puts "body      : #{body}"
    end

    def fib(n)
      n < 2 ? n : fib(n-1) + fib(n-2)
    end  
  end

  class mailbuilder
    def initialize(&block)
      @mail = mail.new
      instance_eval(&block)
    end
    
    attr_reader :mail

    %w(from to subject body).each do |m|
      define_method(m) do |val|
        @mail.send("#{m}=", val)
      end
    end
  end
end

we can then invoke this mailer class as follows to send mail:

mailer.deliver do 
  from    "eki@eqbalq.com"
  to      "jill@example.com"
  subject "threading and forking"
  body    "some content"
end

(note: the source code for this test case is available here on github.)

to establish a baseline for comparison purposes, let’s begin by doing a simple benchmark, invoking the mailer 100 times:

puts benchmark.measure{
  100.times do |i|
    mailer.deliver do 
      from    "eki_#{i}@eqbalq.com"
      to      "jill_#{i}@example.com"
      subject "threading and forking (#{i})"
      body    "some content"
    end
  end
}

this yielded the following results on a quad-core processor with mri ruby 2.0.0p353:

15.250000   0.020000  15.270000 ( 15.304447)

multiple processes vs. multiple threads

there is no “one size fits all” answer when it comes to deciding whether to use multiple processes or to multithread your application. the table below summarizes some of the key factors to consider.

examples of ruby solutions that use multiple processes:

• resque : a redis-backed ruby library for creating background jobs, placing them on multiple queues, and processing them later.
• unicorn : an http server for rack applications designed to only serve fast clients on low-latency, high-bandwidth connections and take advantage of features in unix/unix-like kernels.

examples of ruby solutions that use multithreading:

• sidekiq : a full-featured background processing framework for ruby. it aims to be simple to integrate with any modern rails application and much higher performance than other existing solutions.
• puma : a ruby web server built for concurrency.
• thin : a very fast and simple ruby web server.

multiple processes

before we look into multithreading options, let’s explore the easier path of spawning multiple processes.

in ruby, the fork() system call is used to create a “copy” of the current process. this new process is scheduled at the operating system level, so it can run concurrently with the original process, just as any other independent process can. ( note: fork() is a posix system call and is therefore not available if you are running ruby on a windows platform.)

ok, so let’s run our test case, but this time using fork() to employ multiple processes:

puts benchmark.measure{
  100.times do |i|
    fork do     
      mailer.deliver do 
        from    "eki_#{i}@eqbalq.com"
        to      "jill_#{i}@example.com"
        subject "threading and forking (#{i})"
        body    "some content"
      end
    end
  end
  process.waitall
}

( process.waitall waits for all child processes to exit and returns an array of process statuses.)

this code now yields the following results (again, on a quad-core processor with mri ruby 2.0.0p353):

0.000000   0.030000  27.000000 (  3.788106)

not too shabby! we made the mailer ~5x faster by just modifying a couple of lines of code (i.e., using fork() ).

don’t get overly excited though. although it might be attempting to use forking since it’s an easy solution for concurrency, it has a major drawback which is the amount of memory that it will consume. forking is somewhat expensive, especially if a copy-on-write (cow) is not utilized by the ruby interpreter that you’re using. if your app uses 20mb of memory, for example, forking it 100 times could potentially consume as much as 2gb of memory!

also, although multithreading has its own complexities as well, there are a number of complexities that need to be considered when using fork() , such as shared file descriptors and semaphores (between parent and child forked processes), the need to communicate via pipes, and so on.

multithreading

ok, so now let’s try to make the same program faster using multithreading techniques instead.

multiple threads within a single process have considerably less overhead than a corresponding number of processes since they share address space and memory.

with that in mind, let’s revisit our test case, but this time using ruby’s thread class:

threads = []

puts benchmark.measure{
  100.times do |i|
    threads << thread.new do     
      mailer.deliver do 
        from    "eki_#{i}@eqbalq.com"
        to      "jill_#{i}@example.com"
        subject "threading and forking (#{i})"
        body    "some content"
      end
    end
  end
  threads.map(&:join)
}

this code now yields the following results (again, on a quad-core processor with mri ruby 2.0.0p353):

13.710000   0.040000  13.750000 ( 13.740204)

bummer. that sure isn’t very impressive! so what’s going on? why is this producing almost the same results as we got when we ran the code synchronously?

the answer, which is the bane of existence of many a ruby programmer, is the global interpreter lock (gil) . thanks to the gil, cruby (the mri implementation) doesn’t really support threading.

the global interpreter lock is a mechanism used in computer language interpreters to synchronize the execution of threads so that only one thread can execute at a time. an interpreter which uses gil will always allow exactly one thread and one thread only to execute at a time , even if run on a multi-core processor. ruby mri and cpython are two of the most common examples of popular interpreters that have a gil.

so back to our problem, how can we exploit multithreading in ruby to improve performance in light of the gil?

well, in cruby, the unfortunate answer is that you’re basically stuck and there’s very little that multithreading can do for you.

but if you have the option of using a version other than cruby, you can use an alternative ruby implementation such as jruby or rubinius , since they don’t have a gil and they do support real parallel threading.

to prove the point, here are the results we get when we run the exact same threaded version of the code as before, but this time run it on jruby (instead of cruby):

43.240000   0.140000  43.380000 (  5.655000)

now we’re talkin’!

but…

threads ain’t free

the improved performance with multiple threads might lead one to believe that we can just keep adding more threads – basically infinitely – to keep making our code run faster and faster. that would indeed be nice if it were true, but the reality is that threads are not free and so, sooner or later, you will run out of resources.

let’s say, for example, that we want to run our sample mailer not 100 times, but 10,000 times. let’s see what happens:

threads = []

puts benchmark.measure{
  10_000.times do |i|
    threads << thread.new do     
      mailer.deliver do 
        from    "eki_#{i}@eqbalq.com"
        to      "jill_#{i}@example.com"
        subject "threading and forking (#{i})"
        body    "some content"
      end
    end
  end
  threads.map(&:join)
}

boom! i got an error with my os x 10.8 after spawning around 2,000 threads:

can't create thread: resource temporarily unavailable (threaderror)

as expected, sooner or later we start thrashing or run out of resources entirely. so the scalability of this approach is clearly limited.

thread pooling

fortunately, there is a better way; namely, thread pooling.

a thread pool is a group of pre-instantiated, reusable threads that are available to perform work as needed. thread pools are particularly useful when there are a large number of short tasks to be performed rather than a small number of longer tasks. this prevents having to incur the overhead of creating a thread a large number of times.

a key configuration parameter for a thread pool is typically the number of threads in the pool. these threads can either be instantiated all at once (i.e., when the pool is created) or lazily (i.e., as needed until the maximum number of threads in the pool has been created).

when the pool is handed a task to perform, it assigns the task to one of the currently idle threads. if no threads are idle (and the maximum number of threads have already been created) it waits for a thread to complete its work and become idle and then assigns the task to that thread.

so, returning to our example, we’ll start by using queue (since it’s a thread safe data type) and employ a simple implementation of the thread pool:

require “./lib/mailer” require “benchmark” require ‘thread’

pool_size = 10

jobs = queue.new

10_0000.times{|i| jobs.push i}

workers = (pool_size).times.map do
  thread.new do
    begin      
      while x = jobs.pop(true)
        mailer.deliver do 
          from    "eki_#{x}@eqbalq.com"
          to      "jill_#{x}@example.com"
          subject "threading and forking (#{x})"
          body    "some content"
        end        
      end
    rescue threaderror
    end
  end
end

workers.map(&:join)

in the above code, we started by creating a jobs queue for the jobs that need to be performed. we used queue for this purpose since it’s thread-safe (so if multiple threads access it at the same time, it will maintain consistency) which avoids the need for a more complicated implementation requiring the use of a mutex .

we then pushed the ids of the mailers to the job queue and created our pool of 10 worker threads.

within each worker thread, we pop items from the jobs queue.

thus, the life-cycle of a worker thread is to continuously wait for tasks to be put into the job queue and execute them.

so the good news is that this works and scales without any problems. unfortunately, though, this is fairly complicated even for our simple example case.

celluloid

thanks to the ruby gem ecosystem, much of the complexity of multithreading is neatly encapsulated in a number of easy-to-use ruby gems out-of-the-box.

a great example is celluloid, one of my favorite ruby gems. celluloid framework is a simple and clean way to implement actor-based concurrent systems in ruby. celluloid enables people to build concurrent programs out of concurrent objects just as easily as they build sequential programs out of sequential objects.

in the context of our discussion in this post, i’m specifically focusing on the pools feature, but do yourself a favor and check it out in more detail. using celluloid you’ll be able to build multithreaded programs without worrying about nasty problems like deadlocks, and you’ll find it trivial to use other more sophisticated features like futures and promises.

here’s how simple a multithreaded version of our mailer program is using celluloid:

require "./lib/mailer"
require "benchmark"
require "celluloid"

class mailworker
  include celluloid

  def send_email(id)
    mailer.deliver do 
      from    "eki_#{id}@eqbalq.com"
      to      "jill_#{id}@example.com"
      subject "threading and forking (#{id})"
      body    "some content"
    end       
  end
end

mailer_pool = mailworker.pool(size: 10)

10_000.times do |i|
  mailer_pool.async.send_email(i)
end

clean, easy, scalable, and robust. what more can you ask for?

background jobs

of course, another potentially viable alternative, depending on your operational requirements and constraints would be to employ background jobs . a number of ruby gems exist to support background processing (i.e., saving jobs in a queue and processing them later without blocking the current thread). notable examples include sidekiq , resque , delayed job , and beanstalkd .

for this post, i’ll use sidekiq and redis (an open source key-value cache and store).

first, let’s install redis and run it locally:

brew install redis
redis-server /usr/local/etc/redis.conf

with out local redis instance running, let’s take a look at a version of our sample mailer program ( mail_worker.rb ) using sidekiq:

require_relative "../lib/mailer"
require "sidekiq"

class mailworker
  include sidekiq::worker
  
  def perform(id)
    mailer.deliver do 
      from    "eki_#{id}@eqbalq.com"
      to      "jill_#{id}@example.com"
      subject "threading and forking (#{id})"
      body    "some content"
    end  
  end
end

we can trigger sidekiq with the mail_worker.rb file:

sidekiq  -r ./mail_worker.rb

and then from irb :

⇒  irb
>> require_relative "mail_worker"
=> true
>> 100.times{|i| mailworker.perform_async(i)}
2014-12-20t02:42:30z 46549 tid-ouh10w8gw info: sidekiq client with redis options {}
=> 100

awesomely simple. and it can scale easily by just changing the number of workers.

another option is to use sucker punch , one of my favorite asynchronous ror processing libraries. the implementation using sucker punch will be very similar. we’ll just need to include suckerpunch::job rather than sidekiq::worker , and mailworker.new.async.perform() rather mailworker.perform_async() .

conclusion

high concurrency is not only achievable in ruby, but is also simpler than you might think.

one viable approach is simply to fork a running process to multiply its processing power. another technique is to take advantage of multithreading. although threads are lighter than processes, requiring less overhead, you can still run out of resources if you start too many threads concurrently. at some point, you may find it necessary to use a thread pool. fortunately, many of the complexities of multithreading are made easier by leveraging any of a number of available gems, such as celluloid and its actor model.

another way to handle time consuming processes is by using background processing. there are many libraries and services that allow you to implement background jobs in your applications. some popular tools include database-backed job frameworks and message queues.

forking, threading, and background processing are all viable alternatives. the decision as to which one to use depends on the nature of your application, your operational environment, and requirements. hopefully this article has provided a useful introduction to the options available.