Document Celery integration

[beniwohli]

We should document the automatic integration of Celery when using Django or Flask, as well as document how to manually instrument Celery when it is used in standalone mode.

[violetina]

Any ETA for this ? I fail to enable this. We use structlog with rabbitmq handler. With logstash as shipper. The celery worker uses boto3, requests, amp and elasticsearch. ES as results backend too. We don't use django . Some APIs using flask with apm enabled. Just the celery task instrumentation is what is missing..

[arocketman]

I think documenting the standalone mode would be great, any points to understand how-to right now looking at the source code?

[basepi]

Sorry for the late reply here! I haven't spent a lot of time with our celery support, but it should be about as easy as running register_instrumentation and register_exception_tracking from our celery integration.

[namoshizun]

+1. Some documentation would be of great help. At least that would help people who are googling for APM Celery integrations more quickly find necessary instructions.

[process0]

Here is a working example.

from elasticapm import Client
from elasticapm.contrib.celery import register_exception_tracking, register_instrumentation


apm_client = Client()

register_instrumentation(apm_client)
register_exception_tracking(apm_client)

[riccardomassullo]

Elasticsearch version 7.12 and APM 7.12

Until we added the instrument() call, no useful data was sent to us.
Following the answer in this thread https://discuss.elastic.co/t/running-apm-for-standalone-celery/158693/2 worked for us.

So, from the linked thread, a working example:

from elasticapm import Client, instrument
from elasticapm.contrib.celery import register_exception_tracking, register_instrumentation

instrument()
apm_client = Client(server_url="http://localhost:8200")  # adapt settings as needed
register_exception_tracking(apm_client)
register_instrumentation(apm_client)

[coolacid]

Just a followup here. I've implemented the code above and get events from my celery workers all the way down. ie:

Flask -> Celery Job -> Celery Sub Job

However, if I look at the jobs at the First Celery Job level, I don't see the Sub Jobs like I do at the Flask level. Is it because the trace parent is the Flask Job only? It doesn't cascade?

[basepi]

It's been awhile since I've been in the celery code, but yes, I think that the sub jobs must be tied to the Flask traceparent.

[coolacid]

It would be so much nicer if the parent ids cascaded down instead of always being top level.

[namoshizun]

Hmmm not sure if this might help, but you can add a parent_span_id in the task headers to correlate tasks specifically. See #1500