Home

workerpattern

worker pattern using rabbit MQ, celery and MongoDB

1. Install RabbitMQ
2. setup rabbitMQ for celery
   http://docs.celeryproject.org/en/latest/getting-started/brokers/rabbitmq.html

• create user and set permissions for virtual host

• create virtual host

  abbitmqctl add_user puneet welcome1
  rabbitmqctl add_vhost feedvhost
  rabbitmqctl set_user_tags puneet dev
  rabbitmqctl set_permissions -p feedvhost puneet "." "." ".*"

• start rabbit mq server : rabbitmq-server

• to monitor http://localhost:15672 , can create a user with monitor role to monitor queues

3. create virtual env for your project and install celery,pymongo,PyYAML
   virtualenv vworkpatt
   source source vworkpatt/bin/activate
   pip install celery pymongo PyYAML

4. If you are using pyCharm use the virtual env just created

5. run rabbit mq : rabbitmq-server

6. rn mongodb : mongod

7. start celery :
   celery -A tasks.feedtask.celery worker -Q usage_queue --loglevel=info

8. start flask app
   python3 app.py

Post Feeds : http://0.0.0.0:4001/Usagefeed

[{

    "custid": "101",
    "deviceid": "Dev101",
    "id" : "2342545562456254626"
},{
        "custid": "102",
        "deviceid": "Dev102",
        "id" : "23425455624562dd54626"
    },{
    "custid": "101",
    "deviceid": "Dev103",
            "id" : "234254556fgsfgbhs2456254626"
},{
        "custid": "105",
        "deviceid": "Dev102",
                "id" : "rere2342545562456254626"
    }]

Get job status : http://0.0.0.0:4001/Usagefeed/261d6625-d18e-4945-85da-8f89ff388cf6

• SUCCESS, RECEIVED OR STARTED

9. To check mongo collection for all status: db.celery_taskmeta.find({}).sort({"date_done":-1})

9 . freeze env to generate requirements.txt
pip freeze > requirements.txt

10. To run celery with concurrency option
    celery -A tasks.feedtask.celery worker -Q usage_queue --loglevel=info --concurrency=20

11. for scaling
    celery -A tasks.feedtask.celery worker -Q usage_queue --loglevel=info --autoscale=10,3

References:
http://flask.pocoo.org/docs/0.11/patterns/celery/
http://docs.celeryproject.org/en/latest/getting-started/brokers/rabbitmq.html