Using Faktory with Python

2019-01-08

Faktory is my new polyglot background job system, allowing any programming language to use background jobs. I’ve documented how to use Faktory with Ruby and Go as I’m an expert in both languages. Today I wanted to step outside my comfort zone and try Faktory with a language I don’t know: Python. Let’s see how easy it is for me, a noob, to get Python jobs running with Faktory!

We need three pieces for any background job system:

The client pushes jobs to the server.
The server (i.e. Faktory) manages the queues and jobs.
The worker pulls jobs from the server and executes them.

Server Setup

Most importantly, you’ll need the faktory server installed:

brew tap contribsys/faktory
brew install faktory

On macOS, easy. See Installation if you are on Linux.

Python Setup

First thing I did was install a modern Python with brew install python, which got me Python 3.7.2.

$ brew install python
$ pip install faktory
Successfully installed faktory-0.4.0

We install the faktory package, which is the Python client/worker library provided by cdrx/faktory_worker_python. We’ll use it to push and fetch jobs.

Client Script

We’ll create a very simple script to create a job every second. Here we create an adder job with two random integers for arguments:

# fclient.py
import faktory
import random
import time

time.sleep(1)

with faktory.connection() as client:
    while True:
        client.queue('adder', args=(random.randint(0,1000), random.randint(0,1000)))
        time.sleep(1)

Worker Script

The worker is the long-running process which fetches jobs from Faktory and executes them. We register the set of job types we know about and then wait for jobs from Faktory.

# fworker.py
from faktory import Worker
import time
import logging
logging.basicConfig(level=logging.INFO)

time.sleep(1)

def adder(x, y):
    logging.info("%d + %d = %d", x, y, x + y)

w = Worker(queues=['default'], concurrency=1)
w.register('adder', adder)
w.run()

The Result

Finally I created a Procfile which runs all three parts and used foreman to run it:

$ gem install foreman
$ cat Procfile
faktory: /usr/local/bin/faktory
client: /usr/local/bin/python3.7 ./fclient.py
worker: /usr/local/bin/python3.7 ./fworker.py
$ foreman start

The output:

$ foreman start
10:37:21 faktory.1 | started with pid 7319
10:37:21 client.1  | started with pid 7320
10:37:21 worker.1  | started with pid 7321
10:37:21 faktory.1 | Faktory 0.9.3
10:37:21 faktory.1 | Copyright © 2019 Contributed Systems LLC
10:37:21 faktory.1 | Licensed under the GNU Public License 3.0
10:37:21 faktory.1 | I 2019-01-02T18:37:21.055Z Initializing redis storage at /Users/mikeperham/.faktory/db, socket /Users/mikeperham/.faktory/db/redis.sock
10:37:21 faktory.1 | I 2019-01-02T18:37:21.084Z Web server now listening at localhost:7420
10:37:21 faktory.1 | I 2019-01-02T18:37:21.084Z PID 7319 listening at localhost:7419, press Ctrl-C to stop
10:37:22 worker.1  | INFO:faktory.worker:Registered task: adder
10:37:22 worker.1  | INFO:faktory.connection:Connecting to localhost:7419
10:37:22 worker.1  | INFO:faktory.worker:Queues: default
10:37:22 worker.1  | INFO:faktory.worker:Labels: python
10:37:22 worker.1  | INFO:root:943 + 720 = 1663
10:37:23 worker.1  | INFO:root:131 + 259 = 390
10:37:24 worker.1  | INFO:root:272 + 304 = 576
10:37:25 faktory.1 | I 2019-01-02T18:37:25.090Z Retries processed 2 jobs
10:37:25 worker.1  | INFO:root:418 + 86 = 504
10:37:26 worker.1  | INFO:root:818 + 56 = 874
10:37:27 worker.1  | INFO:root:619 + 237 = 856
10:37:28 worker.1  | INFO:root:926 + 93 = 1019
10:37:29 worker.1  | INFO:root:481 + 812 = 1293
10:37:30 worker.1  | INFO:root:391 + 224 = 615

It works! We are creating a job in our Python client process, sending it to Faktory which distributes it to our Python worker process. Our Python client could be a Django application or anything else. In this way you can scale your job processing across many, many machines and workers.

Now while it is running, open your browser to http://localhost:7420 and check out the nice dashboard as your jobs process. With the dashboard you can see jobs which have recently failed and are awaiting retry. Got a bug in your code? The worker will catch the exception and report the failure to Faktory so it can retry the job later.

If you’re a Python developer, I hope this piques your curiousity to try Faktory. Remember Faktory’s advantage is that you can push and pull jobs with any programming language. Want to create jobs in PHP and process them in Python? No problem! Check out all the different languages supported today.