Intro : Celery and MongoDB

This post serves as more of a tutorial to get a  Hello World up and running while using Celery and MongoDB as the broker . Celery has great documentation but they are  in snippets  across multiple pages and nothing that shows a full working example of  using Celery  with MongoDB which might be helpful for new users .

Install All Required Packages

Packages related to Celery.

pip install celery
pip install -U celery-with-mongodb

Ensure that you have MongoDB installed and running. 10gen have great documentation to get you up and running with MongoDB in  no time.

In this tutorial we will only use a single module  for both the celery application and the tasks.

 

1. Create celeryconfig.py

Lets start by first creating a celeryconfig.py that will store  configuration details that will be used to configure Celery to use MongoDb as the results backend and other settings.

2.Create tasks.py(Celery Application and Worker)

 3. Now that we have our application and configs created lets  start it up!

celery -A tasks worker --loglevel=info

Screen Shot 2013-06-15 at 2.46.53 PM

 

4. Now lets create some work items/tasks to be processed.  You can easily create a script , but we used the CLI  in this example.

Celery Tasks

 5. Lets look on the console of the worker to see the jobs submitted.

As you can see tasks were obtained and processed

Screen Shot 2013-06-15 at 2.51.33 PM

 

6. If you are curious about your MongoDB backend . Lets take a look at the collections that will be used to store the results.

Screen Shot 2013-06-15 at 3.02.30 PM

 

Screen Shot 2013-06-15 at 3.03.01 PM

Thats how you get a simple Hello World up and running with MongoDB.  Will be doing a advance blog post where we use Celery and MongoDB with proper production configurations.

 

 

 

4 comments

  1. Sam   •  

    I am new to Celery. I am trying this tutorial to tryout Celery with MongoDB. I have below versions of needed packages,

    celery==3.1.18
    celery-with-mongodb==3.0
    kombu==3.0.26
    pymongo==3.0.3

    And I have MongoDB version v2.6.11. On trying the command celery -A tasks worker –loglevel=info, I am getting below error,

    ERROR/MainProcess] Unrecoverable error: ConfigurationError(‘Unknown option use_greenlets’,)
    Traceback (most recent call last):
    File “/lib/python3.4/site-packages/kombu/transport/virtual/__init__.py”, line 789, in create_channel
    return self._avail_channels.pop()
    IndexError: pop from empty list

    During handling of the above exception, another exception occurred:


    File “/lib/python3.4/site-packages/pymongo/common.py”, line 452, in validate
    value = validator(option, value)
    File “/lib/python3.4/site-packages/pymongo/common.py”, line 100, in raise_config_error
    raise ConfigurationError(“Unknown option %s” % (key,))
    pymongo.errors.ConfigurationError: Unknown option use_greenlets

    AND sometime I see error like,
    KeyError: ‘auto_start_request’

    Please suggest where I am wrong. is there some issue with the compatibility of different versions of packages and MongoDB?

Leave a Reply

Your email address will not be published. Required fields are marked *