[DEV DOCS] Journey of a task - HLG edition

[fjetter]

The distributed documentation offers a Journey of a Task entry outlining how a client.submit is serialized, submitted to the scheduler, handled, computed, etc. up until the task is forgotten.

When using dask collections we are typically not directly using client.submit and this journey might look drastically different, especially when using High Level Graphs.

dask/community#190 (comment)

see also dask/dask#7709

[GenevieveBuckley]

One thing I found difficult about the "Journey of a Task" docs (and other pages) was that it's hard to convert the information in the docs page into a live example to play around with. I'll copy the contents of my previous slack conversation here for reference:

Semi-related question, how do I get a handle for the scheduler object with the update_graph method in the example on the "Journey of a Task" docs page https://distributed.dask.org/en/latest/journey.html

from operator import add
from dask.distributed import Client

client = Client()
x = 2
y = 3
msg = {
    'op': 'update-graph',
    'tasks': {'z': (add, x, y)},
    'keys': ['z']
}
# How do I get the scheduler?
scheduler.update_graph(tasks=msg['tasks'], keys=msg['keys'])

@jacobtomlinson replied with:

Depending on how you created your cluster object the scheduler may or may not be in your python process (it could be remote somewhere).In the specific example you shared it will be local to you, so you can find it at client.cluster.scheduler. But in cases where it is remote you have to interact via the RPC which you can find at client.scheduler_comm.

So I think you would need to do await client.scheduler_comm.update_graph()

Which helped a little bit, but I still got stuck. Here's me again:

I think it has to be client.cluster.scheduler.update_graph(tasks=msg['tasks'], keys=msg['keys'])

• the update_graph method lives on the scheduler object (in this example, on client.cluster.scheduler)
• the result I'm getting from the update method is None, I don't think we should await that like a coroutine

This seems closer, but not quite right. I'm getting a serialization error from the update_graph method.

from operator import add
from dask.distributed import Client

client = Client()

msg = {
    'op': 'update-graph',
    'tasks': {'z': (add, 2, 3)},
    'keys': ['z']
}

client.cluster.scheduler.update_graph(tasks=msg['tasks'], keys=msg['keys'])  # this gives me a serialization error

It's possible these docs are for broad explanation only, and should not be used to construct simple examples?