Date formatting changed slightly with Utf8json serializer

[rickardp]

NEST/Elasticsearch.Net version:
master

Description of the problem including expected versus actual behavior:

Not sure if this is an acceptable change or not, but since the introduction of Utf8json the date/time format changed slightly (more fraction digits representing milliseconds).

Steps to reproduce:

1. Run unit tests
2. Test ShouldRoundtripDateTimeAndDateTimeOffsetWithSameKindAndOffset fails. The value is now 1999-01-01T01:01:01.0010000+05:00 instead of 1999-01-01T01:01:01.001+05:00 as it used to be.

Again, this might be an acceptable change, in which case the unit test should be fixed.

[Mpdreamz]

Possibly relates to #3802

I think the change itself is fine, does this change cause any problems for you @rickardp ?

An offshoot issue here is that we don't run Tests.Reproduce from our command line build, will create a PR for that now.

[rickardp]

The change is probably fine for me. I just noticed this when investigating if we can switch serializers (we still use Newtonsoft but want to switch to improve performance).

[russcam]

Thanks for opening @rickardp.

This change is a result of moving to Utf8Json and the use of ISO8601DateTimeFormatter for DateTime and ISO8601DateTimeOffsetFormatter for DateTimeOffset. The implementation favours performance over trimming trailing zeroes.

Json.NET's IsoDateTimeConverter is used in 6.x to serialize DateTime, which uses .ToString() and the format pattern "yyyy'-'MM'-'dd'T'HH':'mm':'ss.FFFFFFFK" that will omit fractional values if zero. Similarly, DateTimeOffset are handled by

https://github.com/JamesNK/Newtonsoft.Json/blob/cdf10151d507d497a3f9a71d36d544b199f73435/Src/Newtonsoft.Json/Utilities/DateTimeUtils.cs#L688-L721

whereby trailing 0 fractional values are removed in a while loop.

My preference would be to keep the Utf8Json implementation for performance. A slight aside, but the Utf8Json implementation does help to avoid a small bug in Elasticsearch < 7.1.0 (elastic/elasticsearch#41871), which would have surfaced with Json.NET with fractional digits less than three.

I'll pull the PR in for the test now. Thanks 👍

[russcam]

We discussed this as a team and are going to keep the change as is, and have added a note to the breaking changes docs

Thanks again for raising @rickardp 👍