Developers can customize the JSON serialization contracts of their types

[eiriktsarpalis]

Background and Motivation

Today, System.Text.Json provides two primary mechanisms for customizing serialization on the type level:

1. Customizing the contract via attribute annotations. This is generally our recommended approach when making straightforward tweaks to the contract, however it comes with a few drawbacks:

   1. It requires that the user owns the type declarations they are looking to customize.
   2. It forces repetition when applying the same rule across multiple members. This can be problematic for users defining large sets of DTOs or library authors looking to extend a rule to arbitrary sets of types.
   3. There are inherent limits to the degree of customization achievable. For example, while it is possible to skip a property via JsonIgnoreAttribute, it is impossible to add a JSON property to the contract that doesn't correspond to a .NET property.
   4. For certain users that prefer serializing their domain model directly, introducing System.Text.Json dependencies to the domain layer is considered poor practice.

2. Authoring custom converters. While this mechanism is general-purpose enough to satisfy most customization requirements, it suffers from a couple of problems:

   1. Making straightforward amendments like modifying serializable properties, specifying constructors or injecting serialization callbacks can be cumbersome, since it effectively requires replicating the entire object/collection serialization logic.
   2. Currently, custom converters do not support async/resumable serialization which can result in performance bottlenecks when serializing large objects (we're planning on addressing this independently via Developers should be able to pass state to custom converters. #63795).

API Proposal

namespace System.Text.Json 
{
    public sealed partial class JsonSerializerOptions
    {
        public IJsonTypeInfoResolver TypeInfoResolver { [RequiresUnreferencedCode] get; set; }
    }
}

namespace System.Text.Json.Serialization 
{
    public abstract partial class JsonSerializerContext : IJsonTypeInfoResolver
    {
        // Explicit interface implementation calling into the equivalent JsonSerializerContext abstract method
        JsonTypeInfo System.Text.Json.Serialization.Metadata.IJsonTypeInfoResolver.GetTypeInfo(Type type, JsonSerializerOptions options);
    }
}

namespace System.Text.Json.Serialization.Metadata 
{
    public interface IJsonTypeInfoResolver
    {
        JsonTypeInfo? GetTypeInfo(Type type, JsonSerializerOptions options);
    }

    // Provides the default reflection-based contract metadata resolution
    public class DefaultJsonTypeInfoResolver : IJsonTypeInfoResolver
    {
        [RequiresUnreferencedCode]
        public DefaultJsonTypeInfoResolver() { }

        public virtual JsonTypeInfo GetTypeInfo(Type type, JsonSerializerOptions options);
        
        public IList<Action<JsonTypeInfo>> Modifiers { get; }
    }

    public static class JsonTypeInfoResolver
    {
        public static IJsonTypeInfoResolver Combine(params IJsonTypeInfoResolver[] resolvers);
    }

    // Determines the kind of contract metadata a given JsonTypeInfo instance is customizing
    public enum JsonTypeInfoKind
    {
        None = 0, // Type is either a primitive value or uses a custom converter -- contract metadata does not apply here.
        Object = 1, // Type is serialized as a POCO with properties
        Enumerable = 2, // Type is serialized as a collection with elements
        Dictionary = 3 // Type is serialized as a dictionary with key/value pair entries
    }

    // remove: [System.ComponentModel.EditorBrowsableAttribute(System.ComponentModel.EditorBrowsableState.Never)]
    // note: added abstract
    public abstract partial class JsonTypeInfo
    {
        public Type Type { get; }
        public JsonSerializerOptions Options { get; }

        // The converter instance associated with the type for the given options instance -- this cannot be changed.
        public JsonConverter Converter { get; }

        // The kind of contract metadata we're customizing
        public JsonTypeInfoKind Kind { get; }

        // Untyped default constructor delegate -- deserialization not supported if set to null.
        public Func<object>? CreateObject { get; set; }

        // List of property metadata for types serialized as POCOs.
        public IList<JsonPropertyInfo> Properties { get; }

        // Equivalent to JsonNumberHandlingAttribute annotations.
        public JsonNumberHandling? NumberHandling { get; set; }

        // Factory methods for JsonTypeInfo
        public static JsonTypeInfo<T> CreateJsonTypeInfo<Τ>(JsonSerializerOptions options) { }
        public static JsonTypeInfo CreateJsonTypeInfo(Type type, JsonSerializerOptions options) { }

        // Factory methods for JsonPropertyInfo
        public JsonPropertyInfo CreateJsonPropertyInfo(Type propertyType, string name) { }
    }

    // remove: [EditorBrowsable(EditorBrowsableState.Never)]
    public abstract partial class JsonTypeInfo<T> : JsonTypeInfo
    {
        // Default typed constructor delegate
        public new Func<T>? CreateObject { get; set; }
    }

    // remove: [System.ComponentModel.EditorBrowsableAttribute(System.ComponentModel.EditorBrowsableState.Never)]
    public abstract partial class JsonPropertyInfo
    {
        public JsonSerializerOptions Options { get; }
        public Type PropertyType { get; }
        public string Name { get; set; }

        // Custom converter override at the property level, equivalent to `JsonConverterAttribute` annotation.
        public JsonConverter? CustomConverter { get; set; }

        // Untyped getter delegate
        public Func<object, object?>? Get { get; set; }

        // Untyped setter delegate
        public Action<object, object?>? Set { get; set; }
    
        // Predicate determining whether a property value should be serialized
        public Func<object, object?, bool>? ShouldSerialize { get; set; }

        // Equivalent to JsonNumberHandlingAttribute overrides.
        public JsonNumberHandling? NumberHandling { get; set; }
    }
}

Usage examples

Custom resolver with constructed JsonTypeInfo

static void Main()
{
    MyType[] values = new MyType[] {
        new() { MyStringId = "123", HideName = true, Name = "John" },
        new() { MyStringId = "124", Name = "James" },
        new() { MyStringId = "-1" },
    };
    JsonSerializerOptions options = new();
    options.TypeInfoResolver = new MyCustomResolver();
    string output = JsonSerializer.Serialize(values, options);
    // [{"ID":123},{"ID":124,"Name":"James"},{}]
}

class MyType
{
    public bool HideName { get; set; }
    public string Name { get; set; }
    public string? MyStringId { get; set; } = "-1";
}

class MyCustomResolver : DefaultJsonTypeInfoResolver
{
    public override JsonTypeInfo GetTypeInfo(Type type, JsonSerializerOptions options)
    {
        if (type != typeof(MyType))
            return base.GetTypeInfo(type, options);

        JsonTypeInfo<MyType> jti = JsonTypeInfo.CreateJsonTypeInfo<MyType>(options);
        jti.CreateObject = () => new MyType();

        JsonPropertyInfo<int> propId = jti.CreateJsonPropertyInfo<int>("ID");

        propId.Get = (object o) =>
        {
            MyType m = (MyType)o;
            return m.MyStringId == null ? -1 : ConvertStringIdToInt(m.MyStringId);
        };

        propId.Set = (object o, int val) =>
        {
            MyType m = (MyType)o;
            m.MyStringId = ConvertIntIdToString(val);
        };

        propId.CanSerialize = (object o, int val) =>
        {
            MyType m = (MyType)o;
            return val != -1;
        };

        jti.Properties.Add(propId);

        JsonPropertyInfo<string> propName = jti.CreateJsonPropertyInfo<string>("Name");
        propName.Get = (object o) =>
        {
            MyType m = (MyType)o;
            return m.Name;
        };

        propName.Set = (object o, string val) =>
        {
            MyType m = (MyType)o;
            m.Name = val;
        };

        propName.CanSerialize = (object o, string val) =>
        {
            MyType m = (MyType)o;
            return !m.HideName && m.Name != null;
        };

        jti.Properties.Add(propName);

        return jti;
    }

    private static int ConvertStringIdToInt(string id) => int.Parse(id);
    private static string ConvertIntIdToString(int id) => id.ToString();
}

Adding support for DataMemberAttribute annotations

public class SystemRuntimeSerializationAttributeResolver : DefaultJsonTypeInfoResolver
{
    public override JsonTypeInfo GetTypeInfo(Type type, JsonSerializerOptions options)
    {
        JsonTypeInfo jsonTypeInfo = base.GetJsonTypeInfo(type, options);

        if (jsonTypeInfo.Kind == JsonTypeInfoKind.Object &&
            type.GetCustomAttribute<DataContractAttribute>() is not null)
        {
            jsonTypeInfo.Properties.Clear();

            foreach ((PropertyInfo propertyInfo, DataMemberAttribute attr) in type.GetProperties(BindingFlags.Instance | BindingFlags.Public)
                .Select((prop) => (prop, prop.GetCustomAttribute<DataMemberAttribute>() as DataMemberAttribute))
                .Where((x) => x.Item2 != null)
                .OrderBy((x) => x.Item2!.Order))
            {
                JsonPropertyInfo jsonPropertyInfo = jsonTypeInfo.CreateJsonPropertyInfo(propertyInfo.PropertyType, attr.Name ?? propertyInfo.Name);
                jsonPropertyInfo.Get =
                    propertyInfo.CanRead
                    ? propertyInfo.GetValue
                    : null;

                jsonPropertyInfo.Set = propertyInfo.CanWrite
                    ? (obj, value) => propertyInfo.SetValue(obj, value)
                    : null;

                jsonTypeInfo.Properties.Add(jsonPropertyInfo);
            }
        }

        return jsonTypeInfo;
    }
}

Combining resolver

Doc comparing 3 design variants with recommendation can be found here: https://gist.github.com/krwq/c61f33faccc708bfa569b3c8aebb45d6

JsonPropertyInfo vs JsonPropertyInfo<T> vs JsonPropertyInfo<TDeclaringType, TPropertyType>

We have considered different approaches here and it all boils down to perf of the property setter.
According to simple perf tests run on different combinations of declaring types and property types as well 4 different approaches of setters using setter in form of:
delegate void PropertySetter<DeclaringType, PropertyType>(ref DeclaringType obj, PropertyType val);

proves to be overall fastest. Current implementation would require a bit of work for this to be changed and such support can be added later. Given above we've decided to for a time being support only non-generic PropertyInfo with the slowest setter since such type already exists and corresponding setter would have to be added regardless of choice. In the future PropertyInfo<TDeclaringType, TPropertyType> should be added to support for the fastest possible case.

Here are benchmark results: https://gist.github.com/krwq/d9d1bad3d59ff30f8db2a53a27adc755
Here is the benchmark code: https://gist.github.com/krwq/eb06529f0c99614579f84b69720ab46e

Acceptance Criteria

System.Text.Json already defines a JSON contract model internally, which is surfaced in public APIs via the JsonTypeInfo/JsonPropertyInfo types as opaque tokens for consumption by the source generator APIs. This is a proposal to define an IContractResolver-like construct that builds on top of the existing contract model and lets users generate custom contracts for System.Text.Json using runtime reflection.

Here is a provisional list of settings that should be user customizable in the contract model:

• Object constructors.
• The JSON type of the serialized value: object or array/collection (and possibly primitive values?).
• A list of properties for object types.
• Enumeration/population strategy for array/collection types.
• Serialization callbacks.
• Polymorphic serialization configuration (cf. Developers can use System.Text.Json to serialize type hierarchies securely #63747).
• For the property model:
  • Serialization order.
  • Property name.
  • Getter and setter delegates.
  • Custom converter.
  • Is extension data property.
  • Ignore on serialization/deserialization (cf. [API Proposal]: Add more extensive control over ignoring properties #55781).
  • Number handling.
  • Support modifying (instead of replacing) already initialized properties (cf. Support modifying (rather than replacing) already-initialized properties and fields when deserializing #30258).

Use cases

• JSON/REST transcoding grpc/grpc-dotnet#167
• Support the Specified convention for serializing and deserializing in System.Text.Json #40395
• [API Proposal]: Add more extensive control over ignoring properties #55781
• System.Text.Json: How to serialize only certain properties without using JsonIgnoreAttribute #593
• Add support for MissingMemberHandling to System.Text.Json #37483
• System.Text.Json support to System.Runtime.Serialization #29975
• Each parameter in the deserialization constructor on type 'SpeechCreatedEvent' must bind to an object property or field on deserialization. Each parameter name must match with a property or field on the object. The match can be case-insensitive.' #69547
• Blocking Issue: System.Text.Json and non-public members #31511

Open Questions

• Should we expand the contract model to supporting collection serialization? Is the existing JsonTypeInfo model amenable to supporting collection serialization? Should we consider exposing a JsonCollectionInfo : JsonTypeInfo derived class? Related to Investigate ways to extend deserialization support for arbitrary collection types #38514.

Progress

• Author a working prototype.
• API proposal & review of MVP subset.
• Implementation & tests.
• Add APIs for the remaining items in the Acceptance Criteria.
• Remaining Implementation Work
  • Ensure no size regressions in trimmed applications.
  • System.Text.Json contract customization: not possible to enable deserialization in properties marked JsonIgnoreCondition.Always #71886
  • Reconsider the semantics of JsonProperty.ShouldSerialize null values #71964
  • The JsonSerializerOptions.TypeInfoResolver property should be nullable #71960
  • Add basic regression testing for .NET 6 source generated JsonSerializerContext. #71850
    https://github.com/dotnet/runtime/pull/72044/files#diff-f343f56e41ad406150ed9d35e7548f0d118dd3038c57dff2ff309e5e96331091R588
• Conceptual documentation & blog posts.
  • Improve XML documentation in new APIs.
  • Ensure exception messages are still relevant and up-to-date. Example

cc @steveharter @JamesNK

Tagging subscribers to this area: @dotnet/area-system-text-json
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Background and Motivation

Today, System.Text.Json provides two primary mechanisms for customizing serialization on the type level:

1. Customizing the contract via attribute annotations. This is generally our recommended approach when making straightforward tweaks to the contract, however it comes with a few drawbacks:

   1. It requires that the user owns the type declarations they are looking to customize.
   2. It forces repetition when applying the same rule across multiple members. This can be problematic for users defining large sets of DTOs or library authors looking to extend a rule to arbitrary sets of types.
   3. There are inherent limits to the degree of customization achievable. For example, while it is possible to skip a property via JsonIgnoreAttribute, it is impossible to add a JSON property to the contract that doesn't correspond to a .NET property.
   4. For certain users that prefer serializing their domain model directly, introducing System.Text.Json dependencies to the domain layer is considered poor practice.

2. Authoring custom converters. While this mechanism is general-purpose enough to satisfy most customization requirements, it suffers from a couple of problems:

   1. Making straightforward amendments like modifying serializable properties, specifying constructors or injecting serialization callbacks can be cumbersome, since it effectively requires replicating the entire object/collection serialization logic.
   2. Currently, custom converters do not support async/resumable serialization which can result in performance bottlenecks when serializing large objects.

Proposal

System.Text.Json already defines a JSON contract model internally, which is surfaced in public APIs via the JsonTypeInfo/JsonPropertyInfo types as opaque tokens for consumption by the source generator APIs. This is a proposal to define an IContractResolver-like construct that builds on top of the existing contract model and lets users generate custom contracts for System.Text.Json using runtime reflection.

Here is a provisional list of settings that could be user customizable in the contract model:

• Object constructors.
• The JSON type of the serialized value: object or array/collection (and possibly primitive values?).
• A list of properties for object types.
• Enumeration/population strategy for array/collection types.
• Serialization callbacks.
• Polymorphic serialization configuration (cf. JsonSerializer polymorphic serialization and deserialization support #30083).
• For the property model:
  • Serialization order.
  • Property name.
  • Getter and setter delegates.
  • Custom converter.
  • Is extension data property.
  • Number handling.
  • Support modifying (instead of replacing) already initialized properties (cf. Support modifying (rather than replacing) already-initialized properties and fields when deserializing #30258).

Progress

• Author a working prototype.
• API proposal & review.
• Implementation & tests.
• Conceptual documentation & blog posts.

cc @steveharter @JamesNK

Author:	eiriktsarpalis
Assignees:	eiriktsarpalis
Labels:	area-System.Text.Json, User Story, Priority:0, Cost:L, Team:Libraries
Milestone:	7.0.0