Sidekiq 3.0!

2014-03-28

After tons of work on Sidekiq 2.x for the last 18 months, I decided it was time for some deeper refactoring and cleanup necessitating a major version bump.

Sidekiq 3.0 is the result of three months of hacking, cleanup and community suggestions. There’s an huge amount of stuff in here so hang on to your hats…

Dead Jobs

Before 3.0, jobs which failed all their retries would simply be discarded. Now they are forwarded to a Dead Job Queue. Dead jobs can be manually retried but Sidekiq will ignore them otherwise. Dead jobs will be pruned after six months or 10,000 jobs accumulate so the DJQ doesn’t become a memory leak in Redis.

Along with the new UI features, we’ve got 4 new translations: Chinese, Greek, Swedish and Czech. Did you know the Sidekiq Web UI is available in 15 languages?

Process Heartbeat

Each Sidekiq process now pings Redis with its current state every 5 seconds and the Web UI “Workers” tab has been redesigned to show a real-time view of the Sidekiq processes and the jobs they are processing. This required a large refactoring of Sidekiq’s internal data structures in Redis, which were really based on the original Resque structures I cloned so Sidekiq 1.0 would work with resque-web. These structures were reasonable for Resque’s process-based designed but didn’t work well when a process had many worker threads. Ideally this redesign will fix the “stuck worker” problems that many people saw when their Sidekiq processes crashed.

Note also I’ve completely removed the term “Worker” from Sidekiq where possible in favor of the more clear “Process”, “Thread” or “Job”.

busy tab

Process Lifecycle Events

You can register blocks of code to run at three points during the life of a Sidekiq process:

  1. startup – run after Sidekiq has finished booting but before any work has started
  2. quiet – triggered by the USR1 signal, run when Sidekiq stops accepting new work
  3. shutdown – run as the last code executed by Sidekiq before it exits

It’ll look something like this:

Sidekiq.configure_server do |config|
  config.on(:shutdown) do
    puts "Goodbye cruel world!"
  end
end

Error Handlers

Instead of relying on middleware to catch errors, Sidekiq 3.0 allows you to define one or more global error handlers to process errors generated anywhere within the Sidekiq process. All of the major error services have been contacted and some have already released gem updates to support this new feature. If yours hasn’t, please contact support and politely notify them.

Client Sharding

Sidekiq 2.x has a scalability limit: one Redis server. In practice the limit is greater than 5000 jobs per second on good hardware so to most it wasn’t a big deal but some of the more intense Sidekiq users were hitting that limit. An updated Sidekiq::Client API allows you to specify a Redis connection pool to use, rather than assuming the Sidekiq.redis global pool previously, allowing you to target different types of jobs to different Redis servers. Now you can scale Sidekiq to infinity and beyond!

The API is almost entirely backwards compatible: the only breaking change is in client-side middleware. See the upgrade notes for more detail.

Cleanup

3.0 sees a bunch of deprecated APIs removed and general cleanup:

MRI 1.9 – It seems like just yesterday that MRI 1.9 was state of the art but MRI 2.1.1 was recently released and MRI 2.0 has been very stable for a year now. Sidekiq’s official support policy is to support the current and previous major releases of MRI and Rails and that means MRI 2.0 and 2.1 along with Rails 3.2 and 4.0. To be clear, Sidekiq might work perfectly fine on MRI 1.9 – I just don’t test on it anymore.

Capistrano – we don’t use the official Capistrano recipes at The Clymb and there have never been any tests for them so they’ve proven brittle and hard to maintain. In practice, coming up with a standard deployment recipe that is usable for most people has proven difficult. I strongly recommend using Upstart, runit or another modern init system to manage your ruby daemons rather than “bundle exec’ing” directly from a capistrano recipe. Sidekiq 2.x’s capistrano recipes have moved into the capistrano-sidekiq gem if you wish to continue using them.

connection_pool 2.0 – Eric Hodel aka drbrain was kind enough to contribute a patch to connection_pool making it lazy. Now you can create a pool of size 1000 but it will only establish connections on an as needed basis, meaning you should see a lot less connections in practice, a great upgrade if you are watching your connection counts closely. Fun fact: my first Rubygem was memcache-client, which I took over maintenance from drbrain in 2008!

Conclusion

Now that 3.0 is out, I’ll be turning my focus to Sidekiq Pro 2.0. I’ve already got a few ideas for new features; if you want a feature in Pro, please enter a Sidekiq issue and let me know. Happy ‘kiqing!

For a more comprehensive listing, please see the changelog and upgrade notes.

PS I’m happy to chat with podcasters and bloggers about Sidekiq 3.0, actors/concurrency/threading, sustainable OSS development, OSS business models and the Ruby community. Email me and we’ll work something out.