Celery, Django and Virtualenv playing nice.

A quick post about getting celery daemons to run properly when utilizing virtualenv for your Django projects. The problem I was hitting was so simple and dumb I am upset I didn’t figure it out sooner, but no useful errors were being returned and kept me on a fruitless goose chase.

My problems arose when deploying to our staging server which is running Ubuntu. In a production environment you’ll want your celery task manager to run daemonized. This is made stupidly simple as celery provides all the init scripts you need to accomplish this.

http://github.com/ask/celery/tree/master/contrib/debian/init.d/

Place these scripts in your /etc/init.d/ directory.

All good so far.

Next create a config file for these scripts to use when starting up celery. This is key.

CELERYD_CHDIR="/path/to/virtualenvs/mysite/django_project"

DJANGO_SETTINGS_MODULE="settings"

CELERYD="/path/to/virtualenvs/mysite/bin/python /path/to/virtualenvs/mysite/django_project/manage.py celeryd"

CELERYD_OPTS="-v 2 -B -s celery -E"

CELERYD_LOG_FILE="/path/to/virtualenvs/mysite/logs/celery.log"

CELERYD_USER="www-data"

CELERYD_GROUP="www-data"

Save it here: /etc/default/celeryd.

Most important to note here is the CELERYD value. When using a virtual environment it must be activated when running commands utilizing library instances installed in that virtualenv. Since you don’t have control over activating the environment in automated processes such as this you give it the path to the python binary that lives in your virtualenv and then the path to the manage.py followed by the celerdy command.

Remember this is specific to virtualenv setup. Otherwise you could just use celeryd for the CELERYD config item.

Now it’s just a matter of running /etc/init.d/celeryd start and you’re off to the races! — or at least running background processes in your Django project.

Useful Links

Great tip on altering settings for unittest.

I’d come across some manual overrides on a per-test basis, but they weren’t helpful for project wide settings used in multiple tests such as what database to use.

http://seanhayes.name/2010/01/09/test-database-django/

Create a test_settings.py module and override whatever you want. Throw the following code at the bottom of your main settings.py module and you’re good to go. Check the link if you’d like a full explanation.

if 'test' in sys.argv:
    try:
        from test_settings import *
    except ImportError:
        pass

I am using this to utilize sqlite for testing which yields a HUGE speed boost.

Scaling a Django App

One of the most concise overviews on scaling I’ve ever scene. If you’ve ever had trouble wrapping your head around the different aspects of scaling a web app this is a great read.