Improve dataclass type hinting

[cretz]

Is your feature request related to a problem? Please describe.

Our use of https://github.com/konradhalas/dacite is too primitive. It only supports to/from the object itself, not collections, optional, etc.

Describe the solution you'd like

All of the following should be supported via type hints:

• MyDataClass
• List[MyDataClass], Set[MyDataClass], other collections, etc
• Mapping[str, MyDataClass], Dict[str, MyDataClass], other dicts, etc
• Union[MyDataClass1, MyDataClass2], Union[List[int], Dict[str, MyDataClass]], etc
• Optional[MyDataClass1], Optional[List[Dict[str, MyDataClass]]], etc

Here are the libraries found so far:

• https://github.com/konradhalas/dacite
  • What we currently use
  • No top-level collection, mapping, etc support
  • Could be good base to build off of
• https://github.com/lidatong/dataclasses-json
  • No support for non-decorator/non-mixin use (see issue on their repo numbered 243)
  • Requires marshmallow
• https://github.com/rnag/dataclass-wizard
  • Not the most popular (yet?)
  • Does not require inheritance/mixin use
  • Still requires us to figure out whether it's a dict or list or what not
• https://github.com/lovasoa/marshmallow_dataclass
  • Requires marshmallow (which is a popular library and therefore may clash with other users' marshmallow needs)
  • Seems to be the layer backing dataclasses-json
  • Will still have to determine data types expected

We should probably support customization via instance.to_json()/instance.to_dict() and Class.to_json()/Class.to_dict(). This means those using some of those other libraries will work automatically.

Needs investigation...

[nathanielobrown]

Our use of https://github.com/konradhalas/dacite is too primitive. It only supports to/from the object itself, not collections, optional, etc.

@cretz, just in case you haven't heard of them, I'd like to point out pydantic and fastapi as potentially useful references for this issue.

Pydantic is basically a better dataclasses that's defining feature is really good runtime data validation using types. I use it for serialization/de-serialization with Temporal via a data converter. I find having this runtime checking very useful for catching issues in task processing.

FastAPI is a web framework that uses Pydantic to for serialization/de-serialization. They handle the cases you are referencing here, like taking a list or other generic as an argument. It might be worth a look.

[cretz]

👍 One of my fears w/ pydantic or any of the other high profile modelling dependencies is that I would be carrying that dependency with the SDK. But I will look into using it if adding it isn't too much of a burden on SDK users. It seems the major version is stable enough.

[nathanielobrown]

Yeah, I feel like Pydantic should definitely be optional dependency if included at all. I mostly wanted to point out FastAPI as an example of something that's done exactly what this issue describing.

I could imagine libraries building on top of Temporal (or any task library, really) to provide a FastAPI-like developer experience that's more opinionated and makes more choices for you. But having an extensions/contrib/whatever optional module would be nice too.

[cretz]

Yeah, I feel like Pydantic should definitely be optional dependency if included at all.

I think I have to support dataclass type-hint-based payload conversion out of the box though. But maybe not. Will think on it.

[NullOsama]

Yeah, I feel like Pydantic should definitely be optional dependency if included at all. I mostly wanted to point out FastAPI as an example of something that's done exactly what this issue describing.

I could imagine libraries building on top of Temporal (or any task library, really) to provide a FastAPI-like developer experience that's more opinionated and makes more choices for you. But having an extensions/contrib/whatever optional module would be nice too.

+1 on that.

[cretz]

Unfortunately general-purpose dataclass use with Pydantic appears limited. I have opened discussion at pydantic/pydantic#4318. I will try other options, but I cannot yet find a single Python library that I can give a JSON list of [{"foo": "bar"}] and a type hint of List[MyDataClass] and expect it to do the right thing.

[cretz]

Alternatively, if I don't want to write a full JSON parser using type hints, we could only support TypedDict for now and leave dataclasses alone.

[cretz]

Ok, after internal discussion, we are going to write our own JSON deserializer. I am probably going to borrow from https://github.com/yukinarit/pyserde but remove the requirement to decorate the classes.