Background and Motivation

The current model for authoring custom converters in System.Text.Json is general-purpose and powerful enough to address most serialization customization requirements. Where it falls short currently is in the ability to accept user-provided state scoped to the current serialization operation; this effectively blocks a few relatively common scenaria:

1. Custom converters requiring dependency injection scoped to the current serialization. A lack of a reliable state passing mechanism can prompt users to rebuild the converter cache every time a serialization operation is performed.

2. Custom converters do not currently support streaming serialization. Built-in converters avail of the internal "resumable converter" abstraction, a pattern which allows partial serialization and deserialization by marshalling the serialization state/stack into a state object that gets passed along converters. It lets converters suspend and resume serialization as soon as the need to flush the buffer or read more data arises. This pattern is implemented using the internal JsonConveter<T>.TryWrite and JsonConverter<T>.TryRead methods.

   Since resumable converters are an internal implementation detail, custom converters cannot support resumable serialization. This can create performance problems in both serialization and deserialization:

   • In async serialization, System.Text.Json will delay flushing the buffer until the custom converter (and any child converters) have completed writing.
   • In async deserialization, System.Text.Json will need to read ahead all JSON data at the current depth to ensure that the custom converter has access to all required data at the first read attempt.

   We should consider exposing a variant of that abstraction to advanced custom converter authors.

3. Custom converters are not capable of passing internal serialization state, often resulting in functional bugs when custom converters are encountered in an object graph (cf. ReferenceHandler.IgnoreCycles doesn't work with Custom Converters #51715, Incorrect JsonException.Path when using non-leaf custom JsonConverters #67403, Keep LineNumber, BytePositionInLine and Path when calling JsonSerializer.Deserialize<TValue>(ref reader) #77345)

Proposal

Here is a rough sketch of how this API could look like:

namespace System.Text.Json.Serialization;

public struct JsonWriteState
{
    public CancellationToken { get; }
    public Dictionary<string, object> UserState { get; }
}

public struct JsonReadState
{
    public CancellationToken { get; }
    public Dictionary<string, object> UserState { get; }
}

public abstract class JsonStatefulConverter<T> : JsonConverter<T>
{
     public abstract void Write(Utf8JsonWriter writer, T value, JsonSerializerOptions options, ref JsonWriteState state);
     public abstract T? Read(ref Utf8JsonReader writer, Type typeToConvert, JsonSerializerOptions options, ref JsonReadState state);

      // Override the base methods: implemented in terms of the new virtuals and marked as sealed
      public sealed override void Write(Utf8JsonWriter writer, T value, JsonSerializerOptions options) {}
      public sealed override T? Read(ref Utf8JsonReader writer, Type typeToConvert, JsonSerializerOptions options) {}
}

public abstract class JsonResumableConverter<T> : JsonConverter<T>
{
     public abstract bool TryWrite(Utf8JsonWriter writer, T value, JsonSerializerOptions options, ref JsonWriteState state);
     public abstract bool TryRead(ref Utf8JsonReader writer, Type typeToConvert, JsonSerializerOptions options, ref JsonReadState state, out T? result);

      // Override the base methods: implemented in terms of the new virtuals and marked as sealed
      public sealed override void Write(Utf8JsonWriter writer, T value, JsonSerializerOptions options) {}
      public sealed override T? Read(ref Utf8JsonReader writer, Type typeToConvert, JsonSerializerOptions options) {}
}

public partial class JsonSerializer
{
     // Overloads to existing methods accepting state
     public static string Serialize<T>(T value, JsonSerializerOptions options, ref JsonWriteState state);
     public static string Serialize<T>(T value, JsonTypeInfo<T> typeInfo, ref JsonWriteState state);
     public static T? Deserialize<T>(string json, JsonSerializerOptions options, ref JsonReadState state);
     public static T? Deserialize<T>(string json, JsonTypeInfo<T> typeInfo, ref JsonReadState state);

     // New method groups enabling low-level streaming serialization
     public static bool TrySerialize(T value, JsonTypeInfo<T> typeInfo, ref JsonWriteState state);
     public static bool TryDeserialize(string json, JsonTypeInfo<T> typeInfo, ref JsonReadState state);
}

Users should be able to author custom converters that can take full advantage of async serialization, and compose correctly with the contextual serialization state. This is particularly important in the case of library authors, who might want to extend async serialization support for custom sets of types. It could also be used to author top-level async serialization methods that target other data sources (e.g. using System.IO.Pipelines cf. #29902)

Usage Examples

MyPoco value = new() { Value = "value" };
JsonWriteState state = new() { UserState = { ["sessionId"] = "myId" }};
JsonSerializer.Serialize(value, options, state); // { "sessionId" : "myId", "value" : "value" }

public class MyConverter : JsonStatefulConverter<MyPoco>
{
    public override void Write(Utf8JsonWriter writer, MyPoco value, JsonSerializerOptions options, ref JsonWriteState state)
    {
         writer.WriteStartObject();
         writer.WriteString("sessionId", (string)state.UserState["sessionId"]);
         writer.WriteString("value", value.Value);
     }
}

Alternative designs

We might want to consider the viability attaching the state values as properties on Utf8JsonWriter and Utf8JsonReader. It would avoid the need of introducing certain overloads, but on the flip side it could break scenaria where the writer/reader objects are being passed to nested serialization operations.

Goals

• Support custom resumable converters.
• Support custom converters that are passing the serialization state to child converters.
• Support async serialization using data sources other than Stream (à la [API Proposal]: JsonSerializer.TryReadValue(ref Utf8JsonReader) #29902).
• Support users attaching custom state to serialization operations ([API Proposal]: Support Custom Data on JsonSerializerOptions #71718)

Progress

• Author prototype
• API proposal & review
• Implementation & tests
• Conceptual documentation & blog posts.