Let Utf8JsonReader process input with one complete JSON document per line

[mrange]

EDIT see #33030 (comment) for an API proposal

Apologies if this ticket has been created but I missed it or there's a flag that I missed. I looked around but didn't find a good match.

I work in BigData and a common scenario is that we have large files that contains JSON documents separated by newline like so:

{}
[]
{}

This is not a valid JSON document obviously but it would help me a lot if the Utf8JsonReader could be configured to process a sequence of JSON documents like this.

My alternatives are to read line by line (forcing me to parse the data and thus losing performance) or adding extra complexity when iterating over the objects so I can keep track when an object is done and I should continue to the next.

Neither is very attractive to me and IMHO since this is a quite common scenario it would make some sense to add it, especially since the parser support relaxed parsing wrt to trailing commas and comments

[mrange]

Code example of workaround:

open System
open System.IO
open System.Text.Json

[<EntryPoint>]
let main argv =
  let options = JsonReaderOptions ()
  let input = File.ReadAllBytes "bigdata.json"

  let bytes = ReadOnlyMemory input

  let mutable objectsSeen = 0
  let mutable tokensSeen = 0

  for i = 1 to 1000 do
    let rec processAllLines offset =
      let bytes = bytes.Span

      // Advance the span to next available object
      let bytes = bytes.Slice offset
      let reader = Utf8JsonReader (bytes, options)

      objectsSeen <- objectsSeen + 1
      let mutable level = 0
      let mutable cont = true

      // Consume a single line that assumed to contain a single object

      // reader is a ref struct which prevents usage of tail-rec
      //  :-(
      //  Not really a bug in Utf8JsonReader but rather in the F#
      //  story around ref structs
      while cont &&  reader.Read () do
        tokensSeen <- tokensSeen + 1
        match reader.TokenType with
        | JsonTokenType.StartArray
        | JsonTokenType.StartObject -> level <- level + 1
        | JsonTokenType.EndArray
        | JsonTokenType.EndObject ->
          level <- level - 1
          if level = 0 then
            cont <- false
        | _ -> ()

      let offset = offset + int reader.BytesConsumed

      // After consuming the line there's still trailing non
      //  printable lines, consume them in order to be able to
      //  tell if we reached the end of input
      let rec consumeEndOfLine o =
        if o < input.Length && input.[o] < 32uy then
          consumeEndOfLine (o + 1)
        else
          o

      let offset = consumeEndOfLine offset

      // Done?
      if offset < input.Length then
        processAllLines offset
    processAllLines 0

  printfn "Objects seen : %d" objectsSeen
  printfn "Tokens seen : %d" tokensSeen

  0

[layomia]

From @Int32Overflow in #36750

The following JSON file (with multi content) is not supported:

{ "name": "Admin" }{ "name": "Publisher" }

The following exception occurs:
System.Text.Json.JsonReaderException: "'{' is invalid after a single JSON value. Expected end of data. LineNumber: 0 |

In the library Newtsoft Json.NET there was the following property "SupportMultipleContent". What is the solution here?

https://www.newtonsoft.com/json/help/html/ReadMultipleContentWithJsonReader.htm

[Clockwork-Muse]

... at the very least, using File.ReadLines(...) could help alleviate a large portion of your memory issues. What might be better for the workaround would be if there was some sort of File.ReadLinesAsBytes (since then you wouldn't be deserializing into string first) - essentially, something returning IEnumerable<ReadOnlySpan> that looks for newlines (or possibly something stranger, if you wanted to buffer the input).

[angelobreuer]

I have a similar issue to this, and my issue may fit into this issue:

I am reading UTF-8 JSON data encapsulated in JavaScript from a web server. I've created a stream that skips to the JSON data (according to a specific preamble the JSON data is prefixed by). After the JsonDocument was read, I get an exception that there is data after the object's last token.
My current workaround is to save the entire web page in memory (about 2 MiB but I'm only interested in a few KiB) and put it into a JSON reader which works fine, e.g.:

var jsonData = Encoding.UTF8.GetBytes("{\"d\":\"test\"}}}}}"); //  A JSON object with additional data

// Wanted way
using var memoryStream = new MemoryStream(jsonData); // In my case this is a file stream or HTTP content stream
var document = await JsonDocument.ParseAsync(memoryStream); // FAILS!!
var jsonElement = document.RootElement;

// The "normal" way
var document = JsonDocument.Parse(jsonData.AsMemory()); // FAILS!!
var jsonElement = document.RootElement;

// Workaround:
var jsonReader = new Utf8JsonReader(jsonData.AsSpan(), isFinalBlock: true, default);
var jsonElement = (JsonElement)JsonSerializer.Deserialize<object>(ref jsonReader);

In the first and second sample, I get the following exception: System.Text.Json.JsonReaderException : '}' is invalid after a single JSON value. Expected end of data. LineNumber: 0 | BytePositionInLine: 12..

If I try to deserialize the object, it works fine, and additional data is ignored. The aforementioned is a considerable performance loss, as I have to read the web page and data I'm not interested in.

For example, a property AllowAdditionalContent in JsonDocumentOptions would be fine and would allow users to use System.Text.Json more flexibly.

I appreciate any help you can provide.
Angelo Breuer

[Clockwork-Muse]

@angelobreuer -

using var memoryStream = new MemoryStream(jsonData); // In my case this is a file stream or HTTP content stream`

If your stream is already skipping to the start, why not have it close at the end? Change the stream to look for the end of the JSON section, and returns EOF (Read returns 0) when it detects the end of the json section. If you don't have one, write a wrapping stream that does this.

// The "normal" way
var document = JsonDocument.Parse(jsonData.AsMemory()); // FAILS!!

Something similar is possible here, since memory and sequence objects can take a range, allowing you to extract just that portion.

[angelobreuer]

@Clockwork-Muse

If your stream is already skipping to the start, why not have it close at the end? Change the stream to look for the end of the JSON section, and returns EOF (Read returns 0) when it detects the end of the JSON section. If you don't have one, write a wrapping stream that does this.

As a workaround, I've already done this but scanning each "block" of bytes for the specific end sequence where the JSON ends have significant overhead.

I have migrated from Json.NET where the aforementioned was possible without any additional overhead:

using var streamReader = new StreamReader(someHttpResponseStream); // already skipped to start of JSON
using var jsonTextReader = new JsonTextReader(streamReader);
var data = await JObject.LoadAsync(jsonTextReader);

I've also checked other JSON serializers which allow the same (Utf8Json, Json.NET, ServiceStack.Text).

[eiriktsarpalis]

I think SupportMultipleContent-like option would be a useful addition to Utf8JsonReader, but we'd need to write up an API proposal.