JSLT without Jackson

[larsga]

The current JSLT implementation is so tightly bound to Jackson that every value is a Jackson JsonNode, but there are users who want to use JSLT without Jackson. There is an experimental branch with a custom JSLT VM which could be developed further to provide a Jackson-less JSLT.

There are three main challenges here:

• Functions: these have to be reimplemented. Or one might have a core layer based on pure Java types, which the VM and the existing implementation both translate to.
• Input representation: how is the JSON data passed to JSLT in this case?
• Output representation: how is the JSON result returned?

All of these problems can be solved, but it would be good to get some input on to what extent there are people out there who need a JSLT with zero runtime dependencies. And also what requirements these people have for the input/output representation.

Input wanted!

[mrange]

Hi Lars.

We (Schibsted Pulse) are one of the users that would value a JSLT with no dependencies. Without boring everyone with the details I just mention that we run into conflicts between Spark which has a dependency on Jackson and JSLT that has a dependency on Jackson of a different version.

Below are some thoughts around the problem but I don't claim they are well-thought out.

As the JsonNode is exposed in the API using shadowjar to pull in jackson into the jar and rename the namespace don't seem feasable.

An idea I have been toying with is that JSLT provides it's own writable JSON DOM (perhaps functionally identical to Jackson in all ways that matters to JSLT?).

As a convenience to the users there could be a JSLT.Jackson module that most users depend upon that implement JSON => JSLT JSON DOM => JSON transformations.

That way the core JSLT could have no dependencies and users like us that have specific runtime demands have the possiblilty to implement JSON => JSLT JSON DOM => JSON on a specific version on Jackson or using a different parser all together.

Important is that the API still have to be perceived as convenient to use so our maybe specialized needs don't reduce usability for the normal user. As we might be the odd guys here I would be ok with a slightly less convenient API for us as long as we can exclude Jackson and provide our own parsing.

Another option is to provide several version of JSLT with different Jackson version but I am not a fan of that having to provide several versions of an internal library for different versions of scala.

Regards,

Mårten

[jarno-r]

I've made an experiment replacing Jackson with a simple JSON library:
https://github.com/jarno-r/jslt/tree/json-lib/src/main/java/com/schibsted/spt/data/jslt/json
The fork builds, all of the tests pass and the CLI works.

The main JSLT code is mostly unchanged, except for replacing all Jackson classes with their equivalents. I've replaced almost all uses of asXxxx() methods with the equivalent xxxxValue(), leaving only a couple, where the behaviour differs.

The JSON library is a pretty straightforward replacement for the corresponding Jackson classes, because I just wanted to see how much changes to JSLT are needed. I don't think it's very well designed at the moment.

There's no parser. It's currently using Jackson for parsing. I don't see the lack of a parser as a big problem, since a JSON parser is not hard to implement.

I like the idea of a VM for JSLT, but I think the representation of input & output JSON to JSLT is orthogonal to the implementation, so I would focus on it from an API design point of view.

Here are some thoughts about how I was thinking of improving the JSON lib:

1. Make objects & arrays immutable (other classes already are). JSLT is pure functional (or very close), so this makes a lot of sense.
2. Have only a 'double' based number class. JS numbers are 64-bit floats. Int fits in long and in double, so int is not needed. Integers between +- 2**53 are representable in double, which is plenty. Also having integers and floats handled separately makes for weird semantics, since JSON doesn't really distinguish number types.
3. Make all of the classes be interfaces or abstract classes. This allows multiple implementations for performance optimizations and might help retain original representation (i.e. escape sequences for strings & exact representation of numbers.), if that's desirable. (E.g. a string could be represented as a pointer & length to an input byte[] buffer containing a UTF-8 encoded string, which might never need to be converted to a String. )

Jackson allows NumberNodes that are BigDecimal. This is potentially useful with sums of money, but JSLT doesn't currently really support that. Not sure if that needs to be considered.

Jackson also supports other formats, such as YAML. Should it be an objective for JSLT too?

[jarno-r]

There are two other options to having a JSON library within JSLT:

1. Use 'native' types and values, like null, Boolean, Number, String, Map<String, Object> & List.
2. Use a wrapper interface that exposes all of the methods needed to manipulate all kinds of JSON values, without casting to subtypes. E.g. the wrapper would have methods like isNumber(), isObject(), doubleValue(), stringValue(), get(int), get(String), etc (kind of like JsonNode already does). A factory for constructing new values would also need to be provided.

I don't really like the first option, because it would mean passing Objects around and doing a lot of instanceof checks. Also not sure how null would work.

The second option would be nice in that it wouldn't require any copying between Jackson and another format. One potential issue is that it's pretty opaque to JSLT.

[ecerulm]

There is an experimental branch with a custom JSLT VM which could be developed further to provide a Jackson-less JSLT.

What branch is that?

Functions: these have to be reimplemented. Or one might have a core layer based on pure Java types, which the VM and the existing implementation both translate to.

Uhmm, I think it ok to use Jackson internally, as long as you do a shadowJar/FatJar/UberJar where the jackson classes are relocated. The real "problem" is caused by exposing Jackson classes in the JSLT api that is apply(JsonNode), etc.

If you think Jackson is too heavy weight to embed, then I think it's possible to do a separate JavaCC grammar to parse pure JSON and use the resulting AST (or a layer on top of that) as the internal json library. So that functions, etc are implemented in terms of that internal api and not Jacksons.

Input representation: how is the JSON data passed to JSLT in this case?

java.lang.String the api user can serialize / deserialize to String from their favorite json library and in many cases is likely that the original input is already a String

Output representation: how is the JSON result returned?

java.lang.String also. The api use can deserialize the output json string in their favorite json library.

[wdonne]

When I work with JSON I always use JSON-P. This is only a set of interfaces. It works with a Service Provider Interface. The implementation I use is Glassfish. However, you can write your own by implementing javax.json.spi.JsonProvider.
Currently, in order to hide Jackson from the rest of my code, I have wrapped JSLT in this: https://www.javadoc.io/static/net.pincette/pincette-json/1.3/net/pincette/json/Jslt.html.

[larsga]

JSLT without Jackson

A JSLT implementation that does not depend on Jackson may be useful
for:

• Users may need to avoid clashes with other dependencies that
  need different versions of Jackson.

• Potentially better performance by using a more efficient JSON
  representation.

• Support processing binary formats such as Protobuf or Avro for
  higher performance.

However, there are also a potential downside, in that we may be forced
to implement JSLT more than once. We want to avoid code duplication as
far as we can.

There are a number of different approaches that could be taken, with
different trade-offs.

Define JSLT JsonValue interfaces

We could define a set of Java interfaces to represent JSON values,
then rewrite the JSLT implementation in terms of those. There could
then be an implementation wrapping Jackson nodes in this interface,
and another native implementation with its own parser.

The downside is that performance for Jackson users would probably
suffer a little. Exactly how much is not clear. Most likely it would
not be possible to get any of the potential performance benefits with
this approach.

(I see @jarno-r has tried implementing this. It would be very interesting
to get some performance measurements to help us see what the impact
is.)

Drop-in Jackson replacement

Another possibility would be to implement the Jackson JsonNode
interface in a separate artifact that provides a parser and the
methods and classes that JSLT needs.

The only downside to this would be that we probably could not get the
potential performance benefits.

Also, it probably would not help those who have dependencies that
require a different Jackson version from the one JSLT requires.

Packed JSON representation

We could make a completely different type of JSON representation:
basically mainly ints in arrays. Experiments have been made with
this, showing potential for performance improvements, but without
showing any immediate improvement.

The downside here would be that we end up with two JSLT
implementations: one with Jackson and one without. It's possible that
many of the function implementations could be reused, however.

Generated Java code with adapters

We could make a JSLT implementation that generates either Java source
code or JVM bytecode representing the JSLT logic. This has already
been tried (JVM bytecode) and got a 20% speedup on the first attempt.

If we let the core logic be performed on values that are String,
boolean, long, and so on, much of the JSLT implementation would be
representation-independent.

Different representations of objects and arrays could then be catered
to using adapters that generate different code for touching them,
depending on what representation the user wants to use.

Note that this might also be used to support protobuf/avro input.

There are a few complications that might make this more difficult than
it sounds, but it may be worth at least exploring in greater detail,
because if it does work there don't seem to be any downsides.

Tentative conclusion

It's too early to pick an approach, but it seems clear that both
defining own JSLT interfaces for JSON and code generation should be
explored. Once that's been done we may be in a better position to make
a choice.

[larsga]

@jarno-r It would be interesting if you could do a benchmark comparison of your code with the existing JSLT code. Especially if you could push your code to a forked version of the repo for investigation. Are you interested in doing that?

[larsga]

I wrote my own JSON interfaces and ported JSLT to run on top of that. I did a benchmark where I ran Jackson JSON objects wrapped for this new interface through filters and transforms, and compared it with processing Jackson objects directly. That approach seemed to result in a 7-8% slowdown. Which is not bad at all.

I should note that in this case the output is in the internal JSON representation, and may need to be translated. Or we may need to produce the output as Jackson objects wrapped in this new representation, so that retrieving the Jackson data is trivial.

Users who want to transform serialized JSON input into serialized JSON output probably don't care about Jackson at all. So I should probably do another benchmark for that, because it would involve parsing, transform, plus serialization. It's entirely possible that we could get better performance for this use case.

It would be interesting to hear from users who want to use JSLT with Jackson. What are your use cases? Is it important to get Jackson objects as the output? This is important for me to understand, so that I don't make a new JSLT version that is unusable for you.

[larsga]

The work done so far is available on branch own-json-interfaces

[larsga]

93% of the tests now pass. A good part of what remains is parsing, which is easily fixed.

Transforms with own JSON representation is now marginally slower than with Jackson objects. (Not sure why. Will work on improving this.)

Parsing JSON now seems to have same performance as Jackson.

[biochimia]

I have some concerns about the approach taken in https://github.com/schibsted/jslt/tree/own-json-interfaces to drop the Jackson dependency.

Rolling a custom JSON parser adds significant complexity to the project and is also added risk on the effort to detach JSLT from Jackson. Do we want to maintain a JSON parser in addition to the JSLT parser and language implementation?

I see JSLT as a language to define transformations on a JSON-like object model. On that level, the language doesn't rely on Jackson or JSON. It's the current runtime implementation that picked Jackson and JSON.

Could we move, instead, in a direction where we take the current runtime implementation and split the language from the runtime such that there is a Jackson/JSON-free core language, and a jslt-jackson as its first implementation? Ideally, the split could allow language and implementation to evolve somewhat separately.