====== Celery - Distributed Task Queue ======
http://docs.celeryproject.org/en/latest/
===== Instalation =====
- Install RabbitMQ:
aptitude install rabbitmq-server
pip install librabbitmq
- Install Celery
pip install celery
pip install django-celery
(note: make sure there is not south installed on your system when using django >= 1.7)
if unable to start RabbitMQ and ERROR: epmd error for host address (cannot connect to host/port)
in /var/log/rabbitmq/startup_log, put your hostname in /etc/hosts after localhost
===== Django Integration =====
http://celery.readthedocs.org/en/latest/django/first-steps-with-django.html
- Create project celery config projectname/celery.py:
from __future__ import absolute_import
import os
from celery import Celery
# set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'celertest.settings')
from django.conf import settings
app = Celery('celertest', broker='amqp://guest@localhost//')
# Using a string here means the worker will not have to
# pickle the object when using Windows.
app.config_from_object('django.conf:settings')
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)
app.conf.update(CELERY_RESULT_BACKEND='djcelery.backends.database.DatabaseBackend')
@app.task(bind=True)
def debug_task(self):
print('Request: {0!r}'.format(self.request))
- Create myapp/tasks.py
from __future__ import absolute_import
from celery import shared_task
@shared_task
def add(x, y):
return x + y
@shared_task
def mul(x, y):
return x * y
@shared_task
def xsum(numbers):
return sum(numbers)
@shared_task
def gen_prime(x):
multiples = []
results = []
for i in xrange(2, x+1):
if i not in multiples:
results.append(i)
for j in xrange(i*i, x+1, i):
multiples.append(j)
return results
- Sync django db (./manage.py syncdb or ./manage.py makemigrations && migrate)
- Add djcelery to INSTALLED_APPS and add following lines to project/settings.py:
CELERYBEAT_SCHEDULER = 'djcelery.schedulers.DatabaseScheduler'
CELERY_RESULT_BACKEND = 'djcelery.backends.database:DatabaseBackend'
- Run celery watcher: ./manage.py celeryd -E
- Optionaly: run celerycam and celerymon to enable the monitoring via DB:
./manage.py celerycam
./manage.py celerymon