Celery, Django and Virtualenv playing nice.

A quick post about getting celery daemons to run properly when utilizing virtualenv for your Django projects. The problem I was hitting was so simple and dumb I am upset I didn’t figure it out sooner, but no useful errors were being returned and kept me on a fruitless goose chase.

My problems arose when deploying to our staging server which is running Ubuntu. In a production environment you’ll want your celery task manager to run daemonized. This is made stupidly simple as celery provides all the init scripts you need to accomplish this.

http://github.com/ask/celery/tree/master/contrib/debian/init.d/

Place these scripts in your /etc/init.d/ directory.

All good so far.

Next create a config file for these scripts to use when starting up celery. This is key.

CELERYD_CHDIR="/path/to/virtualenvs/mysite/django_project"

DJANGO_SETTINGS_MODULE="settings"

CELERYD="/path/to/virtualenvs/mysite/bin/python /path/to/virtualenvs/mysite/django_project/manage.py celeryd"

CELERYD_OPTS="-v 2 -B -s celery -E"

CELERYD_LOG_FILE="/path/to/virtualenvs/mysite/logs/celery.log"

CELERYD_USER="www-data"

CELERYD_GROUP="www-data"

Save it here: /etc/default/celeryd.

Most important to note here is the CELERYD value. When using a virtual environment it must be activated when running commands utilizing library instances installed in that virtualenv. Since you don’t have control over activating the environment in automated processes such as this you give it the path to the python binary that lives in your virtualenv and then the path to the manage.py followed by the celerdy command.

Remember this is specific to virtualenv setup. Otherwise you could just use celeryd for the CELERYD config item.

Now it’s just a matter of running /etc/init.d/celeryd start and you’re off to the races! — or at least running background processes in your Django project.

Useful Links

8 Notes/ Hide

  1. markewright reblogged this from derekreynolds
  2. philippw reblogged this from derekreynolds
  3. derekreynolds posted this