This small project explains, how and why to use celery-pool-asyncio and celery-decorator-taskcls libraries.
- Free software: Apache Software License 2.0
There are three entrypoints containing the tasks with the same signatures and doing the same thing, but implemented in a different way. Each of them contain two tasks. I want to explain why task inheritance is so important. Just imagine, you have to implement another 3 crawler tasks. You can compare ways and choose the most pretty to extend.
You can launch a worker by the following command:
celery worker -P celery_pool_asyncio:TaskPool -A step_01_simple_tasks
There is no inheritance here. Just imagine -- you have to change behavior of these tasks. For example, you have to check response status code and retry the http request if response status is not equaled 200. To reach it, you have to change the code of each task. Then imagine, the number of tasks is not 2, but 10, and you have to change each of them. Did you hear about The Last Line Effect? A mistake will definitely be made.
Image source: PVS-Studio corporate article
You can launch a worker by the following command:
celery worker -P celery_pool_asyncio:TaskPool -A step_02_inheritance
It's much better. The same behavior is implemented by the same code. But look to the bottom of file. The Copy-Paste problem is not solved, and there are mistakes will be happened here. The second probles is implementing of one thing (class based task) requires 2 entities -- handler class and launcher function.
Image source: PVS-Studio corporate article
You can launch a worker by the following command:
celery worker -P celery_pool_asyncio:TaskPool -A step_03_taskcls_idea
It looks like the boilerplate problem has been solved
You can also pass default **kwargs like bind = True, timeouts and other via nested class MetaTask, which can be also inherited. More about it is available in the docs and habrahabr post.