Celery - Distributed Task Queue

http://docs.celeryproject.org/en/latest/

1. Install RabbitMQ:

   aptitude install rabbitmq-server
   pip install librabbitmq

2. Install Celery

   pip install celery
   pip install django-celery

(note: make sure there is not south installed on your system when using django >= 1.7)

if unable to start RabbitMQ and

 ERROR: epmd error for host address (cannot connect to host/port)

in /var/log/rabbitmq/startup_log, put your hostname in /etc/hosts after localhost

http://celery.readthedocs.org/en/latest/django/first-steps-with-django.html

1. Create project celery config projectname/celery.py:

   from __future__ import absolute_import
    
   import os
    
   from celery import Celery
    
   # set the default Django settings module for the 'celery' program.
   os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'celertest.settings')
    
   from django.conf import settings
    
   app = Celery('celertest', broker='amqp://guest@localhost//')
    
   # Using a string here means the worker will not have to
   # pickle the object when using Windows.
   app.config_from_object('django.conf:settings')
   app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)
    
   app.conf.update(CELERY_RESULT_BACKEND='djcelery.backends.database.DatabaseBackend')
    
    
   @app.task(bind=True)
   def debug_task(self):
       print('Request: {0!r}'.format(self.request))

2. Create myapp/tasks.py

   from __future__ import absolute_import
    
   from celery import shared_task
    
    
   @shared_task
   def add(x, y):
       return x + y
    
    
   @shared_task
   def mul(x, y):
       return x * y
    
    
   @shared_task
   def xsum(numbers):
       return sum(numbers)
    
    
   @shared_task
   def gen_prime(x):
       multiples = []
       results = []
       for i in xrange(2, x+1):
           if i not in multiples:
               results.append(i)
               for j in xrange(i*i, x+1, i):
                   multiples.append(j)
       return results

3. Sync django db (./manage.py syncdb or ./manage.py makemigrations && migrate)
4. Add djcelery to INSTALLED_APPS and add following lines to project/settings.py:

   CELERYBEAT_SCHEDULER = 'djcelery.schedulers.DatabaseScheduler'
   CELERY_RESULT_BACKEND = 'djcelery.backends.database:DatabaseBackend'

5. Run celery watcher:

   ./manage.py celeryd -E

6. Optionaly: run celerycam and celerymon to enable the monitoring via DB:

   ./manage.py celerycam
   ./manage.py celerymon