Introducing Faktory

2017-10-24

Yesterday I explained how a good background job framework is critical to scaling business applications. Today, meet Faktory.

I’ve spent the last six months building Faktory, a new background job system which aims to bring the best practices developed over the last five years in Sidekiq to every programming language.

The Basics

Faktory is a server daemon which provides a simple API to produce and consume background jobs. Here’s the chatter to create and execute a background job:

> PUSH {"jid":"1238abc712","jobtype":"SomeJob","args":[1,2,"hello"],"queue":"default"}
< OK
> FETCH critical default low
< {"jid":"1238abc712","jobtype":"SomeJob","args":[1,2,"hello"],"queue":"default"}
[...do the work...]
> ACK {"jid":"1238abc712"}
< OK

Jobs are a small JSON hash with a few mandatory keys. Producers PUSH a job to a queue. Consumers FETCH jobs from queues, process them and then either ACK (success) or FAIL the job. Faktory will store and re-enqueue failed jobs over time just as Sidekiq does (these are called “retries”). Networks fail and bugs happen, job retries are critical to a “self-healing” production environment.

To execute jobs, you need a Faktory worker process which can run your business logic in the language of your choice. With the launch today, I’m providing:

• faktory link - high performance background job server
• faktory_worker_ruby link - a Rubygem which can execute jobs fetched from Faktory using Ruby, similar to Sidekiq
• faktory_worker_go link - a library for building a Go-based worker process

How It Works

Many existing job systems provide no persistence or a simple binlog which can be replayed in case of crash.

Faktory goes further and provides the same job persistence, state management and monitoring Web UI that Sidekiq does. It uses Facebook’s high-performance RocksDB embedded datastore internally to persist all job data, queues, error state, etc. It exposes a Web UI (which is similar to Sidekiq’s), allowing you to see the current state of your queues, jobs and workers.

Faktory listens on port 7419 for commands and port 7420 for the Web UI. 7419 for the robot workers, 7420 for the human bosses.

Get Started

The project is brand new so it will take a few weeks to polish the development process. If you are a Gopher, you can build the binary locally with Go 1.9. If you have Vagrant 2.x running, you can run Faktory via the Vagrant box in build/ubuntu. See the Development wiki page for more detail.

We’ll eventually get some install options plugged into the Installation wiki page. Like all new things, it’s rough today but it will get better over time.

FAK

Q. Can I use it in production?
A. It’s a brand new project but only you can determine how risk tolerant you are. I will release 1.0 when the APIs are solid and I feel it is stable.

Q. Does Faktory require Redis?
A. No. Faktory is a standalone 64-bit Linux binary; it needs a Faktory worker process to consume jobs. Redis -> Sidekiq == Faktory -> Faktory worker

Q. Are there other things like Faktory?
A. Yep, beanstalkd, starling, gearman and others. Faktory aims to be more feature-rich and better supported. Many of Faktory’s OSS competitors are “dead” and no longer supported. I am fortunate enough to have both expertise in background jobs and a business model to support Faktory long-term.

Q. What’s going to happen to Sidekiq?
A. Nothing. It’s stable, powerful and fully supported. If you have a Ruby/Rails app, it’s a natural choice for background jobs.

Q. Will you have a commercial version, just like Sidekiq Pro and Enterprise?
A. If Faktory sees good uptake, likely yes in 2018. Several features like periodic jobs, batches and unique jobs would fit very well into a “Faktory Pro”.

Q. Can I help?
A. Absolutely. Faktory’s GitHub repo is contribsys/faktory; try it out and leave feedback. Build a worker library in your language of choice. I’m fluent in Ruby/Go but not much beyond that.

Q. Can Faktory be provisioned and managed as a SaaS?
A. I have no plans to do so myself but I imagine this would be useful to many. I would be happy to chat privately with people interested in offering Faktory as a service, Heroku add-on, etc.

Q. Man, I love the q in Sidekiq. Why no q?
A. Because “Faqtory” sounds like software for building this FAQ and a multi-billion dollar conglomerate has the worldwide trademark for “Worq”.

Q. Where can I ask further questions?
A. Since you’ve read to the bottom, you get top sekret access to the contribsys/faktory Gitter chat room. I’ll hang out there when I can. Stop by and say hi!