[DEVEX-222] Add built-in auto-serialization

[oskardudycz]

Motivation

KurrentDB users should be able to quickly feel efficient in using basic operations like appending new messages, reading them or subscribing to notifications about them. The current API is not fulfilling it as it doesn't provide safe defaults for the most common scenarios like:

• building the state from events,
• appending new events as an outcome of the business logic,
• subscribing to stream etc.

Currently, they need to consider immediately in which order to read the stream, what the deadline is, and what max count to provide. And most importantly, how to serialize data and which serializer to use, as there's none. All of those customisations are needed in the end but shouldn't be thrown immediately on users if they just want to use the default recommended way. In most development environments, you can find the default, mature choices for serialization.

Such code may not be complex; once you get it right, you don't need to change it too often. But if it's stacked with multiple other things you need to keep in mind, then it's easy to miss something. Most importantly, we shouldn't require our users to build their own wrappers around KurrentDB, but we should provide an accessible API to benefit users from using KurrentDB from the get-go.

Solution

The code adds automatic serialization of message payloads. It uses JSON as a default but allows users to implement the binary format themselves. Thanks to that, user can:

• append plain messages ,
• use a simplified Message wrapper that takes data, and optionally metadata and message-id, this hides the unwanted complexity of EventData,
• read messages deserialised automatically. Thanks to that, the user doesn't need to think about which resolved event is correct (event, original event, they can just use Message or DeserializedData fields).
• subscribe to messages deserialised automatically.

To introduce those capabilities, it was necessary to introduce new methods not to make breaking changes. Having that, I benefited from it and reshaped their APIs.

Now, they just take mandatory data as parameters (e.g. stream name when appending or reading from stream) and pack the rest of the optional parameters into the options parameter. It also wraps safe defaults. That's aligned with how Java and Node.js clients are doing.

Thanks to that, users can use such simple APIs as:

var streamName = $"users-{Guid.NewGuid()}";

// Appending
var messages = [new UserRegistered(email, name)];
client.AppendToStreamAsync(streamName, messages);

// Appending with options
var messages = [new UserRegistered(email, name)];
await client.AppendToStreamAsync(
    streamName
    messages, 
    new AppendToStreamOptions{ ExpectedStreamState = StreamState.NoStream }
);

// Reading
List<UserRegistered> readMessages = 
    client.ReadStreamAsync(streamName)
    .Select(e => e.DeserializedData).ToListAsync();

// Reading with options
List<UserRegistered> readMessages = 
    client.ReadStreamAsync(streamName, new ReadStreamOptions { MaxCount = 3 })
    .Select(e => e.DeserializedData).ToListAsync();

// Subscribing
await using var subscription = client.SubscribeToStream(streamName);

// Subscribing with options
await using var subscription = client
    .SubscribeToStream(streamName, new SubscribeToStreamOptions{ Start = FromStream.End})

Next steps

After getting approval for changes in the .NET client, that should be applied accordingly to other clients (starting from Java and Node.js).

This change is foundational for enabling scenarios like getting state from events, running business logic, and appending new events as an outcome.

Details

0. No intentional breaking changes were made.

1. This PR introduces the default System.Text.Json serializer and uses it both for JSON and Binary serialization. Users can change the default serialization and also override it through each operation option.

2. Automatic serialization is only enabled for new methods. Thanks to that, people won't get additional overhead upon migration and will be motivated to migrate to the new methods. I didn't make old methods obsolete, but I'd suggest marking them as [Obsolete("Use the new method x. This method may be removed in future releases", false)]. Such registration won't be marked as a warning but as information, so we won't have friction and confusion. This would also give a hint for existing users that they can use new methods.

3. A basic client-side schema registry was introduced. Currently, it's a simple implementation that is responsible for finding the proper serializer based on the content type and mapping between the message type name and CLR type. Default convention is: {categoryName}-{CLR Type Full Name}.

The CLR name will be of course, specific to .NET, but if customers are also using other clients (e.g. Java, Node.js) then they can override the convention by injecting a custom implementation of the IMessageTypeNamingStrategy interface into settings. They can also define different, more universal conventions that are not based on C# or Java types, but then they'll also need to register those types up front.

Users can register message type mapping upfront or benefit from the automated registration. The automated registration will try to load type from assemblies available in AppDomain. It won't load assemblies. No assembly loading code is provided out of the box or registration by attribute. This can be added to the follow-up PR if needed. We could also use Assembly Qualified Name; then, we'd get assembly loading out of the box.

The Mappings are cached in the type registry to improve performance.

The automatic registration was added to allow automatic use of KurrentDB and diverge convention when needed.

4. General serialization settings can be set through client settings, which include:

• Changing the default content type to octet stream,
• Changing the default System.Text.Json options. That should help to adjust to settings that users are currently using (if they're different from the recommended ones),
• Overriding serializer implementations. User can implement the new ISerializer interface for a specific content type. For now, I haven't provided a serializer for Protobuf and Avro, as the PR is already big. This can (and should) be added later.
• Changing type name mapping convention,
• Register type mappings manually. This can help register types manually upfront or those that are not following the default conventions.
• Register CLR message types for the stream category. They will be mapped using the default (or registered) naming conventions.
• Registering the custom metadata type. For now default serializer either takes TraceMetadata or same metadata type for all messages. Serialization will take data as it is. The more nuanced default handling can be provided by the user in the custom IMessageTypeNamingStrategy implementation. We can also expand it in the future.

All of that can also be customized for each operation.

5. The central piece of serialization is MessageSerializer class that orchestrates the serialization code. It does that by:

• resolving message type using the registered in schema registry mapping strategy,
• taking the serializer from the schema registry based on the intended content type,
• (de)serializing message data using it,
• (de)serializing message metadata as JSON

6. Old methods were made of wrappers for new ones. Thanks to that, the old test suite also covers new methods, and there was no need to repeat it. It'll also be easier to drop them in the future. The integration tests were added, covering additional scenarios.

TODO:

• ensuring that the minimum set of types are expected, and the rest are made internally,
• detailed unit tests after getting an agreement on the design.

[VaughnVernon]

Suggestion for refactoring from standard defaults to custom serialization, etc.

(This is probably the wrong place to suggest but I thought I should offer while I read reading @oskardudycz spec.)

• Surgeons, like programmers, hate using anything but cutting-edge tools. Yet, checklists save lives and prevent bugs:

  • https://www.amazon.com/Checklist-Manifesto-How-Things-Right/dp/0312430000/ref=asc_df_0312430000
  • Provide step-by-step checklists in docs

• Tool to convert to common, opinionated approaches (ids, serialization, options, etc... whatever this issue overs with defaults), such as Protobuf, Avro, NoStream, MaxCount, StreamPosition, etc.

[alexeyzimarev]

Hey @VaughnVernon, we'll be updating the docs closer to the new client package release date. There's a lot of work planned for both clients and related docs. We realise that our client docs are between average and bad depending on who's looking and we aim to make clients docs substantially better.

[VaughnVernon]

@alexeyzimarev Sure. I didn't intend to sound critical about the docs in general. I was just thinking about possible improvements to help with the necessary steps to optimize serialization, etc., that @oskardudycz mentioned in the above description and/or comments. With a sort of checklist, developers can follow the "cookbook" so they don't forget an essential step and not understand why it's not working.

[oskardudycz]

@VaughnVernon appologies for the delayed answer. I think that cookbook is a good suggestion. We'll need to think how to include that in the documentation. I think that those changes can make actually easier to explain gradually how KurrentDB works, as now what's default is the recommended way (e.g. autoserialization, deadlines, etc.). So they can just start appending and reading messages. Then we can explain why and when may consider using additional options. For that checklists make a good sense imho.

[VaughnVernon]

@oskardudycz Yes, the default, opinionated way is essential. Up-and-running makes it easier on the supported and the supporters. Then when it's time to get serious about optimizations, the advanced support will help the supported thread 5 to 7 needles.