JsonReaderException when parsing other JSON within scanValuesFromStream

[ski309]

Hello, I just discovered that if I call scanJsonValuesFromStream to parse a file of JSON objects, then try to parse separate JSON from within the lambda, a JsonReaderException is thrown with the message, expected '{', offset: 0x00000000, buf: .... The buffer data shown is in a location within the InputStream.

example:

val aData: String = ???
val in: InputStream = ???
Using.resource(in) { in =>
  scanJsonValuesFromStream[A](in) { a =>
    readFromString[A](aData)
  }
}

[plokhotnyuk]

Hi, @ski309 !

All readFrom... and writeTo... methods aren't re-entrant due to using the same JsonReader or JsonWriter instances shared through the thread local cache except readFromStringReentrant and writeToStringReentrant.

Please try re-entrant ones for your case.

Yet another option is using own instances of JsonReader or JsonWriter but it will require using com.github.plokhotnyuk.jsoniter_scala.core
package to access their package-only visible constructors.

[ski309]

I see... I hadn't noticed those methods. Are there any plans to introduce reentrant methods for byte arrays or input streams? Also, are there any plans to allow access to create our own JsonReader or JsonWriter instances?

[ski309]

I guess I should clarify that my example was simplified. What actually happened in my case was that, within scanValuesFromStream I called a method in another class that happens to read JSON data from elsewhere.

[ski309]

In any case, thank you for your help!

[plokhotnyuk]

Reentrant methods are much expensive to make them used widely. Probably having some additional constructors from runtime configuration will allow to use CPU and memory resources more efficiently with caching them by users.

[plokhotnyuk]

@ski309 A good option for the scanValuesFromStream callback would be passing a parsed message to different thread for handling of parsed message. As example through some blocking queue with limited size for back pressure. What do you think about that?

[ski309]

@plokhotnyuk I think having to pass messages to a different thread would reduce performance in a hot path in my code. For now, I've switched the JSON parsing within the callback back to Jackson, as it was beforehand.

In the future, I might remove the call to scanValuesFromStream entirely, and manually read the stream, and parse objects from the stream via jsoniter an object at a time.

One other suggestion would be to mention in the JsonReaderException that the error might have been caused by non-reentrant code?

Thanks for your help!

[plokhotnyuk]

@ski309 Please peek the latest v2.14.0 release and try scanValuesFromStreamReentrant if it is an acceptable solution for you.

[ski309]

@plokhotnyuk Works great! Thank you Andriy!