Celery - Distributed Task Queue


  1. Install RabbitMQ:
    aptitude install rabbitmq-server
    pip install librabbitmq
  2. Install Celery
    pip install celery
    pip install django-celery

(note: make sure there is not south installed on your system when using django >= 1.7)

if unable to start RabbitMQ and

 ERROR: epmd error for host address (cannot connect to host/port)

in /var/log/rabbitmq/startup_log, put your hostname in /etc/hosts after localhost

Django Integration


  1. Create project celery config projectname/celery.py:
    from __future__ import absolute_import
    import os
    from celery import Celery
    # set the default Django settings module for the 'celery' program.
    os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'celertest.settings')
    from django.conf import settings
    app = Celery('celertest', broker='amqp://guest@localhost//')
    # Using a string here means the worker will not have to
    # pickle the object when using Windows.
    app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)
    def debug_task(self):
        print('Request: {0!r}'.format(self.request))
  2. Create myapp/tasks.py
    from __future__ import absolute_import
    from celery import shared_task
    def add(x, y):
        return x + y
    def mul(x, y):
        return x * y
    def xsum(numbers):
        return sum(numbers)
    def gen_prime(x):
        multiples = []
        results = []
        for i in xrange(2, x+1):
            if i not in multiples:
                for j in xrange(i*i, x+1, i):
        return results
  3. Sync django db (./manage.py syncdb or ./manage.py makemigrations && migrate)
  4. Add djcelery to INSTALLED_APPS and add following lines to project/settings.py:
    CELERYBEAT_SCHEDULER = 'djcelery.schedulers.DatabaseScheduler'
    CELERY_RESULT_BACKEND = 'djcelery.backends.database:DatabaseBackend'
  5. Run celery watcher:
    ./manage.py celeryd -E
  6. Optionaly: run celerycam and celerymon to enable the monitoring via DB:
    ./manage.py celerycam
    ./manage.py celerymon
python/celery.txt · Last modified: 2015/06/22 16:13 by vondra