Exception while deserializing json payload of abstract kotlin class from disk using BasicPolymorphicTypeValidator activateDefaultTyping

[srusanvin]

Hi All,

I have serialized a kotlin data class (Movie) which inherits a abstract class (Programme). If I serialize it as a programme Jsonmapper has difficulty and gives below exception

com.fasterxml.jackson.databind.exc.InvalidDefinitionException: Cannot construct instance of `Programme` (no Creators, like default constructor, exist): abstract types either need to be mapped to concrete types, have custom deserializer, or contain additional type information

When I introduced Polymorphic Deserializations using BasicPolymorphicTypeValidator activateDefaultTyping(). I am getting below exception. Basically library is not able to write the type details of root element. Apart from using a wrapper class to hold my payload, any other idea please.

com.fasterxml.jackson.databind.exc.InvalidTypeIdException: Could not resolve subtype of [simple type, class Programme]: missing type id property '@class'

Serialized payload onto disk

{"lastInSeason":true,"synopsis":["java.util.ArrayList",["my synopsis1"]]}

Library version:
com.fasterxml.jackson.core:jackson-databind:2.17.1

code is here https://github.com/srusanvin/kotlinabstractclassserializer

[cowtowncoder]

Are you sure you want to use "Default Typing", instead of adding @JsonTypeInfo in the shared based class/interface? It is much easier to get working right, generally, and is safer choice security-wise.
Exception you have indicates that polymorphic type id inclusion settings on reading and writing are inconsistent (reader expects As.PROPERTY but writer uses As.ARRAY).

This also looks like it might not be Kotlin specific.

[srusanvin]

Yes, at this moment we would like to evaluate Default Typing, unless we hit a blocker. Regarding security, what I understood is if we use BasicPolymorphicTypeValidator along with allowed/denied sub types, we are good. Also our code is not exposed via POST/PUT APIs but code is more into processing Kafka messages. (I have limited knowledge on the library)
Regarding the code, would like to understand why Jackson lib is not able to write the type information of root class Movie. Just to update, the same code will work fine if we wrap the Movie inside a WarapperPrograme before write and read. Working code is here https://github.com/srusanvin/kotlinpolydeser.git.
Serialized data when we use wrapper is {"programme":{"@class":"Movie","lastInSeason":true,"synopsis":["java.util.ArrayList",["my synopsis1"]]}}
Any suggestion on code, how we can make it work without the WrapperPrograme class

[cowtowncoder]

I suspect that the type of root value without wrapper does not match criteria for Default Typing. This can be resolved by using a wrapper type -- this is, in fact, preferred way and also removes the need to use Default Typing completely.
I agree, security aspect not a problem in your case.

Actually, looking at code: this:

DefaultTyping.OBJECT_AND_NON_CONCRETE

is the problem: Movie is neither java.lang.Object nor abstract class -- it is concrete implementation.
When given root value without separate type, its getClass() is called so your root value is not thought to require polymorphic handling.
You can use different DefaultTyping value like DefaultTyping.NON_FINAL which would then would match.

It is also possible to define explicit type for root value using ObjectWriter, like:

objectMapper().writer().forType(Programme.class) // parent type, abstract
    .writeValueAsString(myMovieValue);

in which case matching is done using given type, not runtime class of value.

Hope this helps.

[srusanvin]

Hey @cowtowncoder, thanks for responding. I am not sure I get that quite clear. I had already tried DefaultTyping.NON_FINAL prior to wrapper class approach. But it didn't work. That is when I moved onto Wrapper. Regarding DefaultTyping.OBJECT_AND_NON_CONCRETE what I understood is, since I am holding the reference of Movie in a Programme type, which is abstract and serializing it as a programme, while deserializing I faced the challenge of

com.fasterxml.jackson.databind.exc.InvalidDefinitionException: Cannot construct instance of `Programme` (no Creators, like default constructor, exist): abstract types either need to be mapped to concrete types, have custom deserializer, or contain additional type information

Code to reproduce this:

fun main(args: Array<String>) {
    val mapper = JsonMapper.builder().disable(MapperFeature.USE_GETTERS_AS_SETTERS)
        .disable(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES)
        .addModule(
            KotlinModule.Builder()
                .configure(KotlinFeature.NullIsSameAsDefault, enabled = true)
                .build()
        )
//        .activateDefaultTyping(
//            BasicPolymorphicTypeValidator.builder().allowIfSubType(Movie::class.java)
//                .allowIfBaseType(java.util.Map::class.java)
//                .allowIfBaseType(java.util.Set::class.java)
//                .build(),
//            ObjectMapper.DefaultTyping.OBJECT_AND_NON_CONCRETE, JsonTypeInfo.As.PROPERTY
//        )
        .build()

    val movie: Programme = Movie(true)
    val serialized = mapper.writeValueAsBytes(movie)
    val path = Files.createTempDirectory("disk-file").apply { toFile() }
    val filePath = path.resolve("my_movie.json")
    write(filePath, serialized, StandardOpenOption.CREATE, StandardOpenOption.TRUNCATE_EXISTING)
    val fileContent = readAllBytes(filePath)
    val readValue = mapper.readValue(fileContent, Programme::class.java)
    println(readValue)
}

Now if I enable the default typing, by unblocking the commented code in above example,

I get the error,

Could not resolve subtype of [simple type, class Programme]: missing type id property '@class'

[cowtowncoder]

Could you just add @JsonTypeInfo on Programme, instead of using Default Typing (or, possibly, in addition)? That would ensure polymorphic handling for Movie or any other subtype of Programme.

But as to:

"since I am holding the reference of Movie in a Programme type, which is abstract and serializing it as a programme, while deserializing I faced the challenge of "

No. Due to Java (JVM) type erasure, declared type of reference is not visible during runtime; so while you may have reference of type Programme, the only thing visible during runtime (when passing value) is ACTUAL type-erased Class, Movie. So what Jackson ObjectMapper sees is an instance of Movie; it has no way of knowing what reference might think type is.

This is the specific difference between given Root Value (instance you give to writeValue() methods), and ones referenced via Classes (Fields, getter Methods) -- references do have declared type, including generic information. This is why Wrapper works so much better: type is determined from Class definition which does not suffer from Type Erasure.

Given this, your choices -- if you want to use DefaultTyping still, are to pass separate type info via ObjectWriter, or try to use DefaultTyping setting that matches actual runtime type.

[cowtowncoder]

Realized my replies are bit long. So, TL;DNR:

The problem is that Type being determined on write is specific concrete subtype (Movie.class) -- since JVM has no visibility into reference type, only compiler knows it -- but when reading target type is super type of Programme. Since specific DefaultTyping option used -- object and non-concrete, former does NOT match (it is concrete), latter does. That problem would be solved by one of:

1. Different DefaultTyping.NON_FINAL
2. Specifying type to use for root value when writing (ObjectReader.forType(Programmer.class))
3. Enabling polymorphic type handling in super type with @JsonTypeInfo (annotations do inherit so this works)

[srusanvin]

Hey @cowtowncoder , at the outset thanks for helping community. But I still have few questions. Regarding using just wrapper to hold the payload without default typing

I suspect that the type of root value without wrapper does not match criteria for Default Typing. This can be resolved by using a wrapper type -- this is, in fact, preferred way and also removes the need to use Default Typing completely.

I hit a new issue here ,

java.lang.ClassCastException: class java.util.LinkedHashMap cannot be cast to class my_entity_fully_qualified

I just wrapped my payload in a class something like

data class WrapperProgramme(val programme: Any){}

Regarding your suggestions,

1. Different DefaultTyping.NON_FINAL

this is without the usage of wrapper class, I hit the same error . basically it is not able to write the type info to root element

com.fasterxml.jackson.databind.exc.InvalidTypeIdException: Could not resolve subtype of [simple type, class com.sky.search.domain.entities.Programme]: missing type id property '@class'

I am yet to evaluate Suggestion # 2, since its an existing project and lot of entities

Specifying type to use for root value when writing (ObjectReader.forType(Programmer.class))

Using annotaiton is my last option, when we accept that default typing is not capable to getting the job done.

However I am really close, until I hit a new issue (but I am optimistic) by using combination of default typing and wrapper class, but getting some issue with jackson-module-kotlin details along with sample is here #814 can you please take a look. Thanks again!

[cowtowncoder]

Ok, so this:

data class WrapperProgramme(val programme: Any){}

would work without default typing only if @JsonTypeInfo (with appropriate settings) was added to "val programme".
Or with Default Typing if java.lang.Object is included (which includes most DefaultTyping cases).

[srusanvin]

Hi @cowtowncoder Thanks ! Indeed, Wrapper class along with DefaultTyping.JAVA_LANG_OBJECT writes the type info of our roor entity. My understanding with DefaultTyping.JAVA_LANG_OBJECT, is that it will add type info for all the properties that are of type objects. Which is not happening. So this is giving trouble for nested objects deserialize. If I change it to DefaultTyping.DefaultTyping.NON_FINAL it writes type to all the nested elements but gives different isse while deserializing as in #814 . But scope of this question was how to write type info for root element, which we alredy achieved. I will accept this answer and consider opening new for nested elements type data. Huge thanks for support.

[cowtowncoder]

Not quite.

First: think of Default Typing as way to "attach @JsonTypeInfo declaration into set of types without adding annotation". This means that DefaultTyping.JAVA_LANG_OBJECT applies ONLY to values with declared type of java.lang.Object. But what that means depends on whether we are talking about (1) Root Values or (2) referenced Values.
I realize you really want it to apply to Root values -- but I try to explain why that is additional work for little gain.

Root values are values directly passed to ObjectMapper methods: they are not referenced by anything.
In that case, declarted type must be obtained by value.getClass() -- this is type-erased class.
And unless you actually create "new Object()", runtime type is NOT java.lang.Object.
Because of this, DefaultTyping.JAVA_LANG_OBJECT will not apply to most root values.
And you may have hard time figuring out suitable setting to apply to all kinds of value types you might pass.

Reference values are things referenced by Root value or other referenced values. Since there is reference via Class field, Method (setter, getter) or Constructor parameter, there is also full static type from that declaration: this may not be actual type of the value (but its super type). F.ex

public class Point {
   public Number x, y;
   public Object value;
}

in this case, declared type of x is java.lang.Number. Its runtime type could be java.lang.Integer.
Declared type of value is java.lang.Object: its runtime type could be anything.
But if either of those was passed as Root Value, their declared types would be taken from runtime type (because there is no declaration -- any source code references are dropped by compiler).

This is why Wrapper type solves many problems: you can specifically force declared type which is then used for applying Default Typing (or @JsonTypeInfo).

But this is also how you can avoid using Default Typing altogether (which I think is generally always to be preferred when possible):

public class Wrapper {
   @JsonTypeInfo(....)
   public Object value; // or whatever is the common base type you need
}

This will force type information to be included just for that property.

But it would not work if some of referenced types also need polymorphic handling.
Such referenced types should have their own @JsonTypeInfo declarations if so; but this may not always be possible for 3rd party types.