Scaling Celery - Sending Tasks To Remote Machines!

Celery:

Celery is a Python package which implements a task queue mechanism with a foucs on real-time processing, while also supporting task scheduling.

It has 3 main components.

Celery Application(or Client): It is responsible for adding tasks to the queue.

Celery Worker(or Server): It is responsible for executing the tasks given to it.

Broker: It is responsible for transporting messages between client and server.

What You Should Know:

You should know basics of Celery and you should be familiar with

creating celery tasks

from celery import Celery

app = Celery('tasks', backend='amqp',
                      broker='amqp://<user>:<password>@<ip>/<vhost>')

@app.task
def add(x, y):
    return x + y

adding tasks to a queue,

add.apply_async(args=[1,2])

and consuming tasks with a worker

celery worker -A my_app -l info

If your tasks doesn't need much system resources, you can setup all of them in the same machine. But, if you have a lot of jobs which consume resources, then you need to spread them out in several machines.

In this tutorial lets move our celery workers into a remote machine keeping client and broker in same machine.

Sending Tasks To Another Machine:

On Machine A:

Install Celery & RabbitMQ.
Configure RabbitMQ so that Machine B can connect to it.

# add new user
sudo rabbitmqctl add_user <user> <password>

# add new virtual host
sudo rabbitmqctl add_vhost <vhost_name>

# set permissions for user on vhost
sudo rabbitmqctl set_permissions -p <vhost_name> <user> ".*" ".*" ".*"

# restart rabbit
sudo rabbitmqctl restart

Create a new file remote.py with a simple task. Here we have broker installed in machine A. So give ip address of machine 1 in broker url option.

from celery import Celery

app = Celery('tasks', backend='amqp',
                      broker='amqp://<user>:<password>@<ip>/<vhost>')

def add(x, y):
     return x + y

Now we have everything setup on machine A. We can now put some tasks in queue.

In [1]: from remote import add

In [2]: add.delay(1, 2)
Out[2]: <AsyncResult: 3eb96a11-aa61-46d3-9b9d-e0e1703438d0>

In [3]: b.delay(2, 3)
Out[3]: <AsyncResult: ec40db1a-a43c-4486-9530-0a3153fe1380>

In [4]: b.delay(3, 4)
Out[4]: <AsyncResult: ca53a4c7-061b-408e-82ee-86c2d43d21a0>

Everything is setup on machine A. Now lets get into machine B.

On Machine B:

Install Celery.
Copy remote.py file from machine A to this machine.
Run a worker to consume the tasks

celery worker -l info -A remote

As soon as you launch the worker, you will receive the tasks you queued up and gets executed immediately.

[2014-11-09 00:01:19,168: INFO/MainProcess] Received task: remote.add[c2d2bb27-ff5f-47da-b2b9-6fb11669ee1a]
[2014-11-09 00:01:19,170: INFO/MainProcess] Received task: remote.add[8daa1a5c-17d0-46dc-9c93-faf7fbeccdd9]
[2014-11-09 00:01:19,172: INFO/MainProcess] Received task: remote.add[79603d15-24f1-43f8-b8b7-525b7cd4b9a2]
[2014-11-09 00:01:19,401: INFO/MainProcess] Task remote.add[8daa1a5c-17d0-46dc-9c93-faf7fbeccdd9] succeeded in 0.226168102003s: 3
[2014-11-09 00:01:19,462: INFO/MainProcess] Task remote.add[c2d2bb27-ff5f-47da-b2b9-6fb11669ee1a] succeeded in 0.286503815001s: 5
[2014-11-09 00:01:19,464: INFO/MainProcess] Task remote.add[79603d15-24f1-43f8-b8b7-525b7cd4b9a2] succeeded in 0.288741396998s: 7

This is just a simple guide on how to send tasks to remote machines.

Depending on your need, you might have to set up a cluster of servers and route tasks accordingly to scale.

Avil Page

Pages