This post serves as more of a tutorial to get a Hello World up and running while using Celery and MongoDB as the broker . Celery has great documentation but they are in snippets across multiple pages and nothing that shows a full working example of using Celery with MongoDB which might be helpful for new users .
Install All Required Packages
Packages related to Celery.
[syntax type=”js”]
pip install celery
pip install -U celery-with-mongodb
[/syntax]
Ensure that you have MongoDB installed and running. 10gen have great documentation to get you up and running with MongoDB in no time.
In this tutorial we will only use a single module for both the celery application and the tasks.
1. Create celeryconfig.py
Lets start by first creating a celeryconfig.py that will store configuration details that will be used to configure Celery to use MongoDb as the results backend and other settings.
2.Create tasks.py(Celery Application and Worker)
3. Now that we have our application and configs created lets start it up!
[syntax type=”html”]celery -A tasks worker –loglevel=info[/syntax]
4. Now lets create some work items/tasks to be processed. You can easily create a script , but we used the CLI in this example.
5. Lets look on the console of the worker to see the jobs submitted.
As you can see tasks were obtained and processed
6. If you are curious about your MongoDB backend . Lets take a look at the collections that will be used to store the results.
Thats how you get a simple Hello World up and running with MongoDB. Will be doing a advance blog post where we use Celery and MongoDB with proper production configurations.
I am new to Celery. I am trying this tutorial to tryout Celery with MongoDB. I have below versions of needed packages,
celery==3.1.18
celery-with-mongodb==3.0
kombu==3.0.26
pymongo==3.0.3
And I have MongoDB version v2.6.11. On trying the command celery -A tasks worker –loglevel=info, I am getting below error,
ERROR/MainProcess] Unrecoverable error: ConfigurationError(‘Unknown option use_greenlets’,)
Traceback (most recent call last):
File “/lib/python3.4/site-packages/kombu/transport/virtual/__init__.py”, line 789, in create_channel
return self._avail_channels.pop()
IndexError: pop from empty list
During handling of the above exception, another exception occurred:
…
…
File “/lib/python3.4/site-packages/pymongo/common.py”, line 452, in validate
value = validator(option, value)
File “/lib/python3.4/site-packages/pymongo/common.py”, line 100, in raise_config_error
raise ConfigurationError(“Unknown option %s” % (key,))
pymongo.errors.ConfigurationError: Unknown option use_greenlets
AND sometime I see error like,
KeyError: ‘auto_start_request’
Please suggest where I am wrong. is there some issue with the compatibility of different versions of packages and MongoDB?
Hi Sam,
It appears that you ran into a bug that was is still open in Celery for pymongo 3+
https://github.com/sukrit007/celery/commit/e758762e51b9048e98e96a23e93583709c11cfe4
https://github.com/celery/celery/issues/2743
You can apply this patch yourself by editing your local path/to/dist-pack/celery/backends/mongodb.py . Be sure to revert any other changes you have made. Let me know you have any other issues.
Also to resolve ConfigurationError: Unknown option auto_start_request
http://api.mongodb.org/python/current/changelog.html#mongoclient-changes
According to the changelog. The options start_request along with others are also no longer used in pymongo 3+ . You will need to edit the same file above and remove these two lines:
self.options.setdefault('max_pool_size', self.max_pool_size)self.options.setdefault('auto_start_request', False)
Or use the version currently in master of the Celery repository. https://github.com/celery/celery/blob/master/celery/backends/mongodb.py
The latest release they have was made on April. The fixes are not yet in the binaries installed with pip
the docs saved to the mongodb seem to be in a BinData format. Where are the returned values from my task?
a typical doc looks like:
(“_id” : “…”,”status” : “SUCCESS”, ‘date_done” : ISODate(…), “traceback” : BinData(…), “result” : BinData(…), children”: Bindata(…) )
This was a simple example to get started. You would have to configure your tasks to save the results into MongoDB. Example you could override after_return in your task and process and save the results to MongoDb from there
http://docs.celeryproject.org/en/latest/reference/celery.app.task.html#celery.app.task.Task.after_return
PS: Sorry about the late reply. Did not get the notification