Automatically serialize/deserialize activity/suborchestrator arguments and return values

[anthonychu]

Looking at the fan out fan in example, there are are a few places where we have to manually serialize/deserialize data that we should eliminate.

When call_activity is called, we automatically serialize the input value to a string (for example, we serialize the int to a str here). However, in the activity function itself, we need to explicitly convert the value back.

When we return a value from an activity, we require manual serialization. And in the orchestrator, we manually deserialize it.

Proposed changes:

• Always deserialize input values for activity functions.
• Always serialize return values for activity functions.
• Always deserialize values returned by the task from call_activity (and the other call_activity and call_suborchestrator variants).
• Always serialize input value in start_new (looks like we already do this)
• Always deserialize get_input

Some questions:

• What do we do with type hints? (It appears we currently ignore the type hint for an activity function input and always pass a string, as demonstrated here)
• Do we support deserialization to custom classes or stick to primitives like str, int, float, list, dict, etc (whatever json.loads supports)?

The changes will allow the orchestrator code to the written like this:

def orchestrator_function(context: df.DurableOrchestrationContext):
    activity_list = yield context.call_activity("GetActivityCount", 5)

    tasks = [context.call_activity("ParrotValue", i) for i in activity_list]

    values = yield context.task_all(tasks)
    message = yield context.call_activity("ShowMeTheSum", values)

    return message

The activity function would then look like this:

def main(value: int) -> list:
    activity_values = [*range(value)]
    return activity_values

We should be able to remove all import json statements from orchestrators and activities except when we're really working with JSON data.

@davidmrdavid @priyaananthasankar

[davidmrdavid]

I agree, thanks for pointing these out.

I'll answer your questions first:

1. We should not ignore type hints. The example you found is a mistake on my part, I'll fix it promptly. I'm surprised it wasn't caught in my testing, I'll investigate.
2. We should probably support custom classes, for the best experience. Let's also figure out what Durable JS does and consider that as a starter.

I'll take this on and start working on them as a soon as next Monday

[priyaananthasankar]

As for point 2: Python functions supports rich types. Would be really cool to have an example with a rich type!

[anthonychu]

I agree. Might be worth splitting this up into two issues... one to add support for built-in types and another to implement rich types support. The first one seems a bit more straightforward that we can add fairly soon. The latter probably involves some design work.

[davidmrdavid]

I'll tackle them in stages, and just open two PRs, one for each.
Let's start with primitives, of course

[davidmrdavid]

@anthonychu @priyaananthasankar
This seems to be an error on the python worker's side. I've talked with Roger and he'll be looking into it. Let's check back in progress perhaps Wednesday next week.

[davidmrdavid]

I got this working, all of it! I need to do some cleaning and the solution required more book-keeping that I would have originally wanted, but that's just because the library parses JSON at many places.

I should be submitting a PR soon!

[davidmrdavid]

@anthonychu
As important deadlines start closing it, I just double checked all of this again and can confirm, to the extend that some preliminary testing can show, that the bullet-points you listed as proposed changes indeed work, so long as custom classes adhere to exporting a static to_json and from_json methods.

However, one edge case that is not implemented is serializing the output of an orchestrator itself. If, for whatever reason, a user were to return a custom class as the output of their orchestration, we would not be able to serialize it and instead we'd receive a non-determinism exception. I think this can be easily fixed, I just haven't prioritized it due to other work.

I believe this should be fine for now, as sub-orchestrations are not implemented yet. Just want to know your thoughts on the priority of this and also want you to know so it can be documented.

[anthonychu]

Thanks. I think initially we can just document support for lists and dicts and wait till we have classes working everywhere before supporting those.

[davidmrdavid]

With sub-orchestrators implemented, this should already work. I still need to test it to exhaustion, but the serialization layer for sub-orchestrators and regular orchestrators is essentially the same, so I expect them to both work at the moment. I'll be closing this in the meantime, since we've already designed a sensible serialization strategy :)