Support the Specified convention for serializing and deserializing in System.Text.Json

[pierslawson]

Background and Motivation

Both the XmlSerializer and @JamesNK 's JSON.Net support the convention that whether a property on a class (say property XYZ) should be serialized or has been deserialized can be controlled/reported through a second property on the class (which is named by convention as XYZSpecified).

See

runtime/src/libraries/System.Private.Xml/src/System/Xml/Serialization/ReflectionXmlSerializationWriter.cs

Line 606 in 6072e4d

string specifiedMemberName = m.Name + "Specified";

https://github.com/JamesNK/Newtonsoft.Json/blob/6b9f467e817854532ea31e6c08abe47c53ac8b5c/Src/Newtonsoft.Json/Serialization/JsonSerializerInternalWriter.cs#L533

https://github.com/JamesNK/Newtonsoft.Json/blob/7217c484e9705b5e76585c8b7fcd489c8e021c23/Src/Newtonsoft.Json.Tests/Serialization/ShouldSerializeTests.cs#L217

This allows any code working with the deserialized data to tell the difference between a property that has a default value because the source document didn't have a value, and a document that did have a value but it happened to equal the default value. It also allows any code working with the serializer to optionally decide whether a particular property should be serialized or not (at run time).

This is extremely useful when trying to make an API backwards compatible. Say you add a property "NumberOfChildren" to a"Person" class. New client applications will send in values for how many children the person has, but old client applications won't. However, if your code cannot tell the difference between a new client application setting the property to zero and an old client just not sending the data at all, you could mistakenly overwrite the NumberOfChildren value stored in your database.

Proposed API

Please have System.Text.Json allow the calling code to optionally (at runtime) decide whether a particular property should be serialized or not on an object by object basis. Also allow the calling code to be able to tell if a property's value is set because it was deserialised or because it is the default value for the property.

Usage Examples

During serialization:

SpecifiedTestClass` c = new SpecifiedTestClass();
c.Name = "James";
c.Age = 27;
c.NameSpecified = false;
var json = JsonSerializer.Serialize(c);
StringAssert.AreEqual(@"{
  ""Age"": 27
}", json);

During deserialization:

string mikeFullDisclosureString = "{\"Name\": \"Mike Person\", \"NumberOfChildren\": \"0\"}";
var mike = JsonSerializer.Deserialize<FamilyDetails>(mikeFullDisclosureString);
Assert.AreEqual(true, mike.NumberOfChildrenSpecified);

Alternative Designs

Maybe tere is already a way to achieve this that I have missed.

Risks

Performance overhead of looking for "Specified" properties. Also, most "Specified" properties need to be ignored either by using the JsonIgnore or a convention to ignore them.

[rubenprins]

We have a use-case where we receive import data in JSON form, where the JSON schema supports many fields, but clients usually only send some fields. It's crucial we know which fields are present and which aren't, and neither defaults nor nulls work for this. With Newtonsoft.Json this was a breeze. But migrating to System.Text.Json is a real headache!

Right now, we can get away by setting the Specified properties in the property setters (lots of code bloat), but it's no longer possible to reserialize/roundtrip back to JSON.

---

This is related to #593 and #50294.

Note that System.Xml and Newtonsoft.Json also support an additional variant via a bool ShouldSerializeXyz() method next to a property Xyz. This method (which is only used for serialization) is treated the same as an XyzSpecified property.

The Specified convention is both for serialization, to emit/omit properties, and deserialization, to flag which properties were present in the payload.

The XyzSpecified convention can largely be emulated by making a property setter also set its buddy Specified property (as long as you don't use AutoMapper et al.), but it's impossible to emulate the Specified convention for serialization via a JsonConverter, as you cannot keep the remainder of the JsonSerializer's functionality without completely reimplementing it (property naming, ordering, property-level converters, null/default handling, cycle detection, etc.)

It looks fairly simple to add this support to ObjectDefaultConverter and JsonPropertyInfo (e.g., making ShouldSerialize a method bool ShouldSerialize(object instance) and handling the Specified setter via the Set action). But everything there is internal and not open to such extensions right now. So not only would you need to reimplement the entire Converter, but also the entire metadata API.

[eiriktsarpalis]

What benefits would such a convention provide over the using existing JsonIgnoreAttribute? Making migration easier presumably?

[rubenprins]

@eiriktsarpalis JsonIgnoreAttribute is global, static, and compile-time only; XyzSpecified is a property, and thus it can instruct the serializer to omit properties on a per-instance basis, and is not tied to defaults or null values (a null valued property is different from an omitted optional property in JSON schema).

Also, XyzSpecified is set during deserialization too, so you can inspect the deserialization result to see whether (optional) properties were present in the JSON payload, without needing to fall back to a JsonDocument.