====== Celery - Distributed Task Queue ====== http://docs.celeryproject.org/en/latest/ ===== Instalation ===== - Install RabbitMQ: aptitude install rabbitmq-server pip install librabbitmq - Install Celery pip install celery pip install django-celery (note: make sure there is not south installed on your system when using django >= 1.7) if unable to start RabbitMQ and ERROR: epmd error for host address (cannot connect to host/port) in /var/log/rabbitmq/startup_log, put your hostname in /etc/hosts after localhost ===== Django Integration ===== http://celery.readthedocs.org/en/latest/django/first-steps-with-django.html - Create project celery config projectname/celery.py: from __future__ import absolute_import import os from celery import Celery # set the default Django settings module for the 'celery' program. os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'celertest.settings') from django.conf import settings app = Celery('celertest', broker='amqp://guest@localhost//') # Using a string here means the worker will not have to # pickle the object when using Windows. app.config_from_object('django.conf:settings') app.autodiscover_tasks(lambda: settings.INSTALLED_APPS) app.conf.update(CELERY_RESULT_BACKEND='djcelery.backends.database.DatabaseBackend') @app.task(bind=True) def debug_task(self): print('Request: {0!r}'.format(self.request)) - Create myapp/tasks.py from __future__ import absolute_import from celery import shared_task @shared_task def add(x, y): return x + y @shared_task def mul(x, y): return x * y @shared_task def xsum(numbers): return sum(numbers) @shared_task def gen_prime(x): multiples = [] results = [] for i in xrange(2, x+1): if i not in multiples: results.append(i) for j in xrange(i*i, x+1, i): multiples.append(j) return results - Sync django db (./manage.py syncdb or ./manage.py makemigrations && migrate) - Add djcelery to INSTALLED_APPS and add following lines to project/settings.py: CELERYBEAT_SCHEDULER = 'djcelery.schedulers.DatabaseScheduler' CELERY_RESULT_BACKEND = 'djcelery.backends.database:DatabaseBackend' - Run celery watcher: ./manage.py celeryd -E - Optionaly: run celerycam and celerymon to enable the monitoring via DB: ./manage.py celerycam ./manage.py celerymon