API for writeable DOM and C# dynamic support

[steveharter]

Background and Motivation

See dotnet/designs#163. This contains various background and API details.

Proposed API

All types live in the existing System.Text.Json assembly.

The raw data from the "ref\System.Text.Json.cs" file:

namespace System.Text.Json.Node
{
    public sealed partial class JsonArray : JsonNode, ICollection<JsonNode?>, IEnumerable<JsonNode?>, IList<JsonNode?>, IEnumerable
    {
        public JsonArray(JsonNodeOptions? options = default(JsonNodeOptions?));
        public JsonArray(JsonNodeOptions options, params JsonNode?[] items);
        public JsonArray(params JsonNode?[] items);
        public int Count { get; }
        public static JsonArray? Create(JsonElement element, JsonNodeOptions? options = default(JsonNodeOptions?));
        bool ICollection<JsonNode?>.IsReadOnly { get; }
        public void Add(JsonNode? item);
        public void Add<T>(T value);
        public void Clear();
        public bool Contains(JsonNode? item);
        public IEnumerator<JsonNode?> GetEnumerator();
        public int IndexOf(JsonNode? item);
        public void Insert(int index, JsonNode? item);
        public bool Remove(JsonNode? item);
        public void RemoveAt(int index);
        void ICollection<JsonNode?>.CopyTo(JsonNode?[]? array, int arrayIndex);
        IEnumerator IEnumerable.GetEnumerator();
        public override void WriteTo(Utf8JsonWriter writer, JsonSerializerOptions? options = null);
    }
    public abstract partial class JsonNode
    {
        internal JsonNode();
        public JsonNode? this[int index] { get; set; }
        public JsonNode? this[string propertyName] { get; set; }
        public JsonNodeOptions? Options { get; }
        public JsonNode? Parent { get; }
        public JsonNode Root { get; }
        public JsonArray AsArray();
        public JsonObject AsObject();
        public JsonValue AsValue();
        public string GetPath();
        public virtual TValue GetValue<[DynamicallyAccessedMembersAttribute(DynamicallyAccessedMemberTypes.PublicConstructors | DynamicallyAccessedMemberTypes.PublicFields | DynamicallyAccessedMemberTypes.PublicProperties)]TValue>();
        public static explicit operator bool(JsonNode value);
        public static explicit operator byte(JsonNode value);
        public static explicit operator char(JsonNode value);
        public static explicit operator DateTime(JsonNode value);
        public static explicit operator DateTimeOffset(JsonNode value);
        public static explicit operator decimal(JsonNode value);
        public static explicit operator double(JsonNode value);
        public static explicit operator Guid(JsonNode value);
        public static explicit operator short(JsonNode value);
        public static explicit operator int(JsonNode value);
        public static explicit operator long(JsonNode value);
        public static explicit operator bool?(JsonNode? value);
        public static explicit operator byte?(JsonNode? value);
        public static explicit operator char?(JsonNode? value);
        public static explicit operator DateTimeOffset?(JsonNode? value);
        public static explicit operator DateTime?(JsonNode? value);
        public static explicit operator decimal?(JsonNode? value);
        public static explicit operator double?(JsonNode? value);
        public static explicit operator Guid?(JsonNode? value);
        public static explicit operator short?(JsonNode? value);
        public static explicit operator int?(JsonNode? value);
        public static explicit operator long?(JsonNode? value);
        [CLSCompliantAttribute(false)]
        public static explicit operator sbyte?(JsonNode? value);
        public static explicit operator float?(JsonNode? value);
        [CLSCompliantAttribute(false)]
        public static explicit operator ushort?(JsonNode? value);
        [CLSCompliantAttribute(false)]
        public static explicit operator uint?(JsonNode? value);
        [CLSCompliantAttribute(false)]
        public static explicit operator ulong?(JsonNode? value);
        [CLSCompliantAttribute(false)]
        public static explicit operator sbyte(JsonNode value);
        public static explicit operator float(JsonNode value);
        public static explicit operator string?(JsonNode? value);
        [CLSCompliantAttribute(false)]
        public static explicit operator ushort(JsonNode value);
        [CLSCompliantAttribute(false)]
        public static explicit operator uint(JsonNode value);
        [CLSCompliantAttribute(false)]
        public static explicit operator ulong(JsonNode value);
        public static implicit operator JsonNode(bool value);
        public static implicit operator JsonNode(byte value);
        public static implicit operator JsonNode(char value);
        public static implicit operator JsonNode(DateTime value);
        public static implicit operator JsonNode(DateTimeOffset value);
        public static implicit operator JsonNode(decimal value);
        public static implicit operator JsonNode(double value);
        public static implicit operator JsonNode(Guid value);
        public static implicit operator JsonNode(short value);
        public static implicit operator JsonNode(int value);
        public static implicit operator JsonNode(long value);
        public static implicit operator JsonNode?(bool? value);
        public static implicit operator JsonNode?(byte? value);
        public static implicit operator JsonNode?(char? value);
        public static implicit operator JsonNode?(DateTimeOffset? value);
        public static implicit operator JsonNode?(DateTime? value);
        public static implicit operator JsonNode?(decimal? value);
        public static implicit operator JsonNode?(double? value);
        public static implicit operator JsonNode?(Guid? value);
        public static implicit operator JsonNode?(short? value);
        public static implicit operator JsonNode?(int? value);
        public static implicit operator JsonNode?(long? value);
        [CLSCompliantAttribute(false)]
        public static implicit operator JsonNode?(sbyte? value);
        public static implicit operator JsonNode?(float? value);
        [CLSCompliantAttribute(false)]
        public static implicit operator JsonNode?(ushort? value);
        [CLSCompliantAttribute(false)]
        public static implicit operator JsonNode?(uint? value);
        [CLSCompliantAttribute(false)]
        public static implicit operator JsonNode?(ulong? value);
        [CLSCompliantAttribute(false)]
        public static implicit operator JsonNode(sbyte value);
        public static implicit operator JsonNode(float value);
        public static implicit operator JsonNode?(string? value);
        [CLSCompliantAttribute(false)]
        public static implicit operator JsonNode(ushort value);
        [CLSCompliantAttribute(false)]
        public static implicit operator JsonNode(uint value);
        [CLSCompliantAttribute(false)]
        public static implicit operator JsonNode(ulong value);
        public static JsonNode? Parse(string json, JsonNodeOptions? nodeOptions = default(JsonNodeOptions?), JsonDocumentOptions documentOptions = default(JsonDocumentOptions));
        public static JsonNode? Parse(ref Utf8JsonReader reader, JsonNodeOptions? nodeOptions = default(JsonNodeOptions?));
        public static JsonNode? Parse(IO.Stream utf8Json, JsonNodeOptions? nodeOptions = default(JsonNodeOptions?), JsonDocumentOptions documentOptions = default(JsonDocumentOptions));
        public static JsonNode? Parse(ReadOnlySpan<byte> utf8Json, JsonNodeOptions? nodeOptions = default(JsonNodeOptions?), JsonDocumentOptions documentOptions = default(JsonDocumentOptions));
        public string ToJsonString(JsonSerializerOptions? options = null);
        public override string ToString();
        public abstract void WriteTo(Utf8JsonWriter writer, JsonSerializerOptions? options = null);
    }
    public partial struct JsonNodeOptions
    {
        private int _dummyPrimitive;
        public bool PropertyNameCaseInsensitive { readonly get; set; }
    }
    public sealed partial class JsonObject : JsonNode, ICollection<KeyValuePair<string, JsonNode?>>, IDictionary<string, JsonNode?>, IEnumerable<KeyValuePair<string, JsonNode?>>, IEnumerable
    {
        public JsonObject(JsonNodeOptions? options = default(JsonNodeOptions?));
        public JsonObject(IEnumerable<KeyValuePair<string, JsonNode?>> properties, JsonNodeOptions? options = default(JsonNodeOptions?));
        public int Count { get; }
        public static JsonObject? Create(JsonElement element, JsonNodeOptions? options = default(JsonNodeOptions?));
        bool ICollection<KeyValuePair<string, JsonNode?>>.IsReadOnly { get; }
        ICollection<string> IDictionary<string, JsonNode?>.Keys { get; }
        ICollection<JsonNode?> IDictionary<string, JsonNode?>.Values { get; }
        public void Add(KeyValuePair<string, JsonNode?> property);
        public void Add(string propertyName, JsonNode? value);
        public void Clear();
        public bool ContainsKey(string propertyName);
        public IEnumerator<KeyValuePair<string, JsonNode?>> GetEnumerator();
        public bool Remove(string propertyName);
        bool ICollection<KeyValuePair<string, JsonNode?>>.Contains(KeyValuePair<string, JsonNode> item);
        void ICollection<KeyValuePair<string, JsonNode?>>.CopyTo(KeyValuePair<string, JsonNode>[] array, int index);
        bool ICollection<KeyValuePair<string, JsonNode?>>.Remove(KeyValuePair<string, JsonNode> item);
        bool IDictionary<string, JsonNode?>.TryGetValue(string propertyName, [System.Diagnostics.CodeAnalysis.NotNullWhen(true)]out JsonNode jsonNode);
        IEnumerator IEnumerable.GetEnumerator();
        public bool TryGetPropertyValue(string propertyName, out JsonNode? jsonNode);
        public override void WriteTo(Utf8JsonWriter writer, JsonSerializerOptions? options = null);
    }
    public abstract partial class JsonValue : JsonNode
    {
        private protected JsonValue(JsonNodeOptions? options = null);
        public static JsonValue? Create<T>(T value, JsonNodeOptions? options = default(JsonNodeOptions?));
        public abstract bool TryGetValue<[DynamicallyAccessedMembersAttribute(DynamicallyAccessedMemberTypes.PublicConstructors | DynamicallyAccessedMemberTypes.PublicFields | DynamicallyAccessedMemberTypes.PublicProperties)]T>([System.Diagnostics.CodeAnalysis.NotNullWhen(true)]out T? value);
    }
}

To support the dynamic keyword in C#, the "inbox" build will also contain:

    public partial class JsonNode : System.Dynamic.IDynamicMetaObjectProvider
    {
        System.Dynamic.DynamicMetaObject System.Dynamic.IDynamicMetaObjectProvider.GetMetaObject(System.Linq.Expressions.Expression parameter) { throw null; }
    }

Non-inbox builds won't support for dynamic (at least for now, pending feedback) since it would require a new package reference to System.Linq.Expressions.

Additions to JsonSerializerOptions:

namespace System.Text.Json
{
    // Determines the type to create for extension data and properties declared as System.Object.
    public enum JsonUnknownTypeHandling
    {
        JsonElement = 0, // Default
        JsonNode = 1, // Create JsonNode*-derived types for System.Object and JsonObject for extension data.
    }

    public partial class JsonSerializerOptions
    {
        public JsonUnknownTypeHandling UnknownTypeHandling {get; set;}
    }
}

Usage Examples

See the referenced design above. Here's a simple deserialize and serialize example:

JsonObject jObject = JsonObject.Parse("{""MyProperty"":42}");
int i = (int) jObject["MyProperty"];
string json = jObject.ToJsonString();

Tagging subscribers to this area: @eiriktsarpalis, @layomia
See info in area-owners.md if you want to be subscribed.

Issue Details

---

<PLACEHOLDER for now, actively being updated>

Background and Motivation

See dotnet/designs#163.

Proposed API

Note these types live in System.Text.Json.Serialization since they depend on the serializer.

namespace System.Text.Json.Serialization
{
    public abstract class JsonNode
    {
        internal JsonNode(); // prevent externally derived types.

        // JsonArray terse syntax support (no JsonArray cast necessary).
        public JsonNode? this[int index] { get; set; }

        // JsonArray perf: avoid creating 'JsonNode' for read-only scenarios.
        public virtual T To<T>(int index);
        public virtual object To(Type type, int index);

        // JsonObject terse syntax support (no JsonObject cast necessary ).
        public virtual JsonNode? this[string propertyName] { get; set; }

        // JsonObject perf: avoid creating 'JsonNode' for read-only scenarios.
        public virtual T To<T>(string propertyName);
        public virtual object To(Type type, string propertyName);

        public System.Text.Json.JsonSerializerOptions? Options { get; }

        // Return the internal value or convert to the provided type as necessary.
        public abstract T To<T>();
        public abstract object To(Type type);
        public abstract bool TryTo<T>(out T? value);
        public abstract bool TryTo(Type type, out object? value);

        // Return the parent node; useful for LINQ.
        // This is set internally once added to a JsonObject or a JsonElement.
        public JsonNode? Parent { get; }

        // This represents the internal value, and no conversion is performed.
        // By default on deserialize this a JsonElement but can be set to any CLR object in edit mode.
        public virtual object Value {get; set;}

        // Convert a non-dynamic node to a dynamic.
        public dynamic ToDynamic();

        // The token type from deserialization; for new instances JsonValueKind.Unspecified.
        public System.Text.Json.JsonValueKind ValueKind { get; }

        // Serialize\deserialize wrappers. These are helpers and thus are an optional API feature.
        // Parse() terminology consistent with Utf8JsonReader\JsonDocument.
        // WriteTo() terminology consistent with Utf8JsonWriter.
        // Also consider naming variants:
        // - Serialize() \ Deserialize()
        // - FromJson() \ ToJson()
        // - ToString() override in addition or to replace "public string Write()"
        public string Write();
        public void WriteTo(Utf8JsonWriter writer);
        public Task WriteToAsync(Stream utf8Json, CancellationToken cancellationToken = default);
        public byte[] WriteUtf8Bytes();

        public static JsonNode? Parse(string? json, JsonSerializerOptions options = null);
        public static JsonNode? Parse(System.Text.Json.Utf8JsonWriter writer, JsonSerializerOptions options = null);
        public static ValueTask<JsonNode?> ParseAsync(Stream utf8Json, CancellationToken cancellationToken = default, JsonSerializerOptions options = null);
        public static JsonNode? ParseUtf8Bytes(ReadOnlySpan<byte> utf8Json, JsonSerializerOptions options = null);
    }

    public class JsonValue : JsonNode
    {
        public JsonValue(object? value, JsonSerializerOptions? options = null);

        // Can be used to obtain any compatible value, including values from custom converters.
        public override T To<T>();
        public override object To(Type type);
        public override bool TryTo<T>(out T? value);
        public override bool TryTo(Type type, out object? value);
    }

    public class JsonObject : JsonNode,
        ICollection<KeyValuePair<string, JsonNode?>>,
        IDictionary<string, JsonNode?>,
        IEnumerable<KeyValuePair<string, JsonNode?>>,
        IEnumerable
    {
        public JsonObject(JsonSerializerOptions? options = null);

        // Can be used to return:
        // - Any compatible dictionary type
        // - POCO including anonymous type or record
        // - JsonElement
        // - string or byte[] (as JSON)
        public override T To<T>();
        public override object To(Type type);
        public override bool TryTo<T>(out T? value);
        public override bool TryTo(Type type, out object? value);

        // Used for performance, avoids creation of 'JsonNode'
        public override T To<T>(string propertyName);
        public override object To(Type type, string propertyName);

        // Interface implementations; the non-common members are implemented explicitly
        public int Count { get; }
        bool ICollection<KeyValuePair<string, JsonNode?>>.IsReadOnly { get; }
        ICollection<string> IDictionary<string, JsonNode?>.Keys { get; }
        ICollection<JsonNode?> IDictionary<string, JsonNode?>.Values { get; }
        public void Add(string key, JsonNode? value);
        public void Clear();
        public bool ContainsKey(string key);
        public IEnumerator<KeyValuePair<string, JsonNode?>> GetEnumerator();
        public bool Remove(string key);
        void ICollection<KeyValuePair<string, JsonNode?>>.Add(KeyValuePair<string, JsonNode> item);
        bool ICollection<KeyValuePair<string, JsonNode?>>.Contains(KeyValuePair<string, JsonNode> item);
        void ICollection<KeyValuePair<string, JsonNode?>>.CopyTo(KeyValuePair<string, JsonNode>[] array, int arrayIndex);
        bool ICollection<KeyValuePair<string, JsonNode?>>.Remove(KeyValuePair<string, JsonNode> item);
        IEnumerator IEnumerable.GetEnumerator();
    }

    public class JsonArray : JsonNode,
        ICollection<JsonNode?>,
        IEnumerable<JsonNode?>,
        IList<JsonNode?>,
        IEnumerable
    {
        public JsonArray(JsonSerializerOptions? options = null);

        // Can be used to return:
        // - Any compatible collection type
        // - JsonElement
        // - string or byte[] (as JSON)
        public override T To<T>();
        public override object To(Type type);
        public override bool TryTo<T>(out T? value);
        public override bool TryTo(Type type, out object? value);

        // Used for performance, avoids creation of 'JsonNode'
        public override T To<T>(string propertyName);
        public override object To(Type type, string propertyName);

        // Interface implementations; the non-common members are implemented explicitly
        public int Count { get; }
        bool ICollection<JsonNode?>.IsReadOnly { get; }
        public void Add(object? item);
        public void Add(JsonNode? item);
        public void Clear();
        public bool Contains(JsonNode? item);
        public IEnumerator<JsonNode?> GetEnumerator();
        public int IndexOf(JsonNode? item);
        public void Insert(int index, JsonNode? item);
        public bool Remove(JsonNode? item);
        public void RemoveAt(int index);
        void ICollection<JsonNode?>.CopyTo(JsonNode?[]? array, int arrayIndex);
        IEnumerator IEnumerable.GetEnumerator();
    }

    // Used to support C# dynamic
    public sealed class JsonDynamicObject : JsonObject, System.Dynamic.IDynamicMetaObjectProvider
    {
        public JsonDynamicObject(JsonSerializerOptions options = null);
        System.Dynamic.DynamicMetaObject System.Dynamic.IDynamicMetaObjectProvider.GetMetaObject(System.Linq.Expressions.Expression parameter);
    }
}

C# dynamic support

In order to enable dynamic-compatible types to be supported by the serializer, EnableDynamicTypes() needs to be called. This is necessary to support the ILLinker being able to remove the assembly reference to the large System.Linq.Expressions assembly when dynamic functionality is not necessary.

namespace System.Text.Json
{
    public partial class JsonSerializerOptions
    {
        public void EnableDynamicTypes();
    }
}

In order to support dynamic internally as well as allowing a consumer to create a new dynamic type the following classes are provided:

namespace System.Text.Json.Serialization
{
    // Used to support C# dynamic
    public sealed  class JsonDynamicArray : JsonArray, System.Dynamic.IDynamicMetaObjectProvider
    {
        public JsonDynamicArray(JsonSerializerOptions options = null);
        System.Dynamic.DynamicMetaObject System.Dynamic.IDynamicMetaObjectProvider.GetMetaObject(System.Linq.Expressions.Expression parameter);
    }

    // Used to support C# dynamic
    public sealed class JsonDynamicValue : JsonValue, System.Dynamic.IDynamicMetaObjectProvider
    {
        public JsonDynamicValue(object? value, JsonSerializerOptions options = null);
        System.Dynamic.DynamicMetaObject System.Dynamic.IDynamicMetaObjectProvider.GetMetaObject(System.Linq.Expressions.Expression parameter);
    }
}

Usage Examples

todo

Alternative Designs

todo

Risks

todo

Author:	steveharter
Assignees:	-
Labels:	api-needs-work, area-System.Text.Json
Milestone:	6.0.0

[Tornhoof]

May I suggest to change Task WriteToAsync(Stream, to ValueTask, just as in ParseAsync?
Maybe also switch the order of cancelationtoken and Options in ParseAsync, having the token at the end is more common.

[TonyValenti]

Yes. Token should always be at the end.

[steveharter]

@Tornhoof

May I suggest to change Task WriteToAsync(Stream, to ValueTask, just as in ParseAsync?
Maybe also switch the order of cancelationtoken and Options in ParseAsync, having the token at the end is more common.

The current design section "Features not proposed in 6.0" states that we don't want to have every permutation that JsonSerializer.Deserialize already has. This is for varying reasons but one is to that the overloads add clutter at least for non-common cases -- e.g. the design has static "Parse()" method on all 4 node types in order to prevent casts, so every time we add another "Parse()" overload it has to go into 4 classes...

A work-around is to use the JsonSerializer.SerializeAsync() directly.

Also note that currently the design does not purport to implement true async read\write behavior with nodes. If this is an important concern, please reply with scenarios. Internally the serializer for non-node scenarios (POCOs and collections) does have true async behavior on both deserialize and serialize meaning async calls to a Stream via ReadAsync() \ WriteAsync() are called often to keep the internal buffer size smallish. However, nodes do not support this -- this means during deserialize the Stream is drained upfront (mostly because JsonElement is used which has that behavior) and since JsonElement doesn't have any existing async-based "write" methods, the existing serializer-based SerializeAsync() must be used that will write to a growable buffer until finished, then write the whole thing to the Stream in one call.