Updating a third party library causes db corruption errors

[unixguru2k]

We see nitrite as one of the core components in our SDK.

Been battling this issue now for a few months.

This happens randomly, when updating a third party library causes read page position errors while attempting to read a repository.

Latest issue popped up while updating to the newest version of admob, db page corruption error occured.

Similar to #111

Any help on this issue would be highly appreciated.

Has anyone been battling similar issues?

If yes, can someone suggest a solution - We are using proguard and nitrite v3.4.3

Cheers
HD

[unixguru2k]

Config Model Class:

[anidotnet]

Please add reproducible code as much as possible.

[unixguru2k]

There is no easy way to reproduce this, but as i mentioned there was no change in the code from the previous version and while performing an update to a third party lib it happened.

Conditions verified:

1. Proguard - Release build (doesn't happen in debug builds or emulator)
2. Admob was updated [nothing else changed in code except the library]

I made a new build, updated it to play store and as soon as i update it on device and try to open the init section barfs out due to db corruption.

Again to clarify, it doesn't happen on each and every release build.

Can you tell me what would be helpful to share in helping us debug this.

[anidotnet]

Initially you have to ensure there is no transitive dependencies of h2 mvstore in your app.. Secondly, you have to double check the proguard rules to see if you are missing anything or not there. This comment here should help you. Lastly, you have to disable compaction of database as mvstore compaction is not very stable. Let me know if you have already tried those.

And also attach the error log here.

[anidotnet]

You could try below proguard rules

-keep class org.diztart.no2.** { *; }
-keep class org.slf4j.** { *; }
-keep class org.h2.** { *; }
-keep class org.objenesis.** { *; }
-keep class com.fasterxml.jackson.** { *; }

-keepnames class * implements java.io.Serializable

-keepclassmembers class * implements java.io.Serializable {
    static final long serialVersionUID;
    private static final java.io.ObjectStreamField[] serialPersistentFields;
    !static !transient <fields>;
    private void writeObject(java.io.ObjectOutputStream);
    private void readObject(java.io.ObjectInputStream);
    java.lang.Object writeReplace();
    java.lang.Object readResolve();
}

If this doesn't succeed, try disabling proguard once to see if it is the root cause of your problem or not.

[unixguru2k]

Thanks @anidotnet - for helping me debug this i appreciate it.

Attaching the proguard entries for the base sdk [screenshow below]

Let me try the other recommendations and get back in a couple of hours.

[unixguru2k]

@anidotnet - when you say remove compact, did you mean compressed() option?
or i should use this .disableAutoCompact()

[anidotnet]

both

[unixguru2k]

Just happened again, i have already implemented your suggestions as mentioned in the previous comments

But after updating couple google libraries namely firebase analytics + firebase messaging in our core sdk, the below issue happened again.

Stack trace below:

2022-03-22 07:13:04.998 24743-24743/? W/System.err: java.lang.IllegalStateException: Unable to read the page at position 549755860946 [1.4.200/6]
2022-03-22 07:13:04.999 24743-24743/? W/System.err: at org.h2.mvstore.DataUtils.newIllegalStateException(DataUtils.java:1)
2022-03-22 07:13:04.999 24743-24743/? W/System.err: at org.h2.mvstore.MVStore.readPage(MVStore.java:6)
2022-03-22 07:13:04.999 24743-24743/? W/System.err: at org.h2.mvstore.MVMap.readPage(MVMap.java:1)
2022-03-22 07:13:04.999 24743-24743/? W/System.err: at org.h2.mvstore.Page$NonLeaf.getChildPage(Page.java:3)
2022-03-22 07:13:04.999 24743-24743/? W/System.err: at org.h2.mvstore.Page.get(Page.java:4)
2022-03-22 07:13:04.999 24743-24743/? W/System.err: at org.h2.mvstore.MVMap.get(MVMap.java:2)
2022-03-22 07:13:04.999 24743-24743/? W/System.err: at org.h2.mvstore.MVMap.get(MVMap.java:1)
2022-03-22 07:13:04.999 24743-24743/? W/System.err: at org.h2.mvstore.MVMap.containsKey(MVMap.java:1)
2022-03-22 07:13:04.999 24743-24743/? W/System.err: at sc.a.containsKey(NitriteMVMap.java:1)
2022-03-22 07:13:04.999 24743-24743/? W/System.err: at org.dizitart.no2.internals.IndexMetaService.g(IndexMetaService.java:1)
2022-03-22 07:13:04.999 24743-24743/? W/System.err: at oc.i.k(NitriteService.java:2)
2022-03-22 07:13:04.999 24743-24743/? W/System.err: at oc.d.J(DefaultNitriteCollection.java:2)
2022-03-22 07:13:04.999 24743-24743/? W/System.err: at qc.b.w(DefaultObjectRepository.java:7)
2022-03-22 07:13:04.999 24743-24743/? W/System.err: at qc.b.y(DefaultObjectRepository.java:2)
2022-03-22 07:13:04.999 24743-24743/? W/System.err: at qc.b.(DefaultObjectRepository.java:4)
2022-03-22 07:13:04.999 24743-24743/? W/System.err: at qc.f.a(RepositoryFactory.java:1)
2022-03-22 07:13:04.999 24743-24743/? W/System.err: at ic.g.A(Nitrite.java:5)
2022-03-22 07:13:04.999 24743-24743/? W/System.err: at com.machaao.android.sdk.Machaao.getDB(Machaao.java:37)
2022-03-22 07:13:04.999 24743-24743/? W/System.err: at com.machaao.android.sdk.Machaao.getDB(Machaao.java:4)
2022-03-22 07:13:04.999 24743-24743/? W/System.err: at com.machaao.android.sdk.Machaao.getConfigRepository(Machaao.java:1)
2022-03-22 07:13:04.999 24743-24743/? W/System.err: at com.machaao.android.sdk.Machaao.init(Machaao.java:56)
2022-03-22 07:13:04.999 24743-24743/? W/System.err: at com.machaao.android.sdk.Machaao.initAsync(Machaao.java:1)
2022-03-22 07:13:04.999 24743-24743/? W/System.err: at messenger.x.chat.bot.messenger.Application.onCreate(Application.java:5)
2022-03-22 07:13:04.999 24743-24743/? W/System.err: at android.app.Instrumentation.callApplicationOnCreate(Instrumentation.java:1223)
2022-03-22 07:13:04.999 24743-24743/? W/System.err: at android.app.ActivityThread.handleBindApplication(ActivityThread.java:6762)
2022-03-22 07:13:04.999 24743-24743/? W/System.err: at android.app.ActivityThread.access$1500(ActivityThread.java:256)
2022-03-22 07:13:04.999 24743-24743/? W/System.err: at android.app.ActivityThread$H.handleMessage(ActivityThread.java:2091)
2022-03-22 07:13:04.999 24743-24743/? W/System.err: at android.os.Handler.dispatchMessage(Handler.java:106)
2022-03-22 07:13:04.999 24743-24743/? W/System.err: at android.os.Looper.loopOnce(Looper.java:201)
2022-03-22 07:13:04.999 24743-24743/? W/System.err: at android.os.Looper.loop(Looper.java:288)
2022-03-22 07:13:04.999 24743-24743/? W/System.err: at android.app.ActivityThread.main(ActivityThread.java:7870)
2022-03-22 07:13:04.999 24743-24743/? W/System.err: at java.lang.reflect.Method.invoke(Native Method)
2022-03-22 07:13:04.999 24743-24743/? W/System.err: at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:548)
2022-03-22 07:13:04.999 24743-24743/? W/System.err: at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1003)
2022-03-22 07:13:04.999 24743-24743/? W/System.err: Caused by: java.lang.IllegalArgumentException: Could not deserialize [-84, -19, 0, 5, 115, 114, 0, 53, 111, 114, 103, 46, 100, 105, 122, 105, 116, 97, 114, 116, 46, 110, 111, 50, 46, 105, 110, 116, 101, 114, 110, 97, 108, 115, 46, 73, 110, 100, 101, 120, 77, 101, 116, 97, 83, 101, 114, 118, 105, 99, 101, 36, 73, 110, 100, 101, 120, 77, 101, 116, 97, 113, 36, -56, -29, -79, -56, 93, 69, 2, 0, 3, 76, 0, 5, 105, 110, 100, 101, 120, 116, 0, 24, 76, 111, 114, 103, 47, 100, 105, 122, 105, 116, 97, 114, 116, 47, 110, 111, 50, 47, 73, 110, 100, 101, 120, 59, 76, 0, 8, 105, 110, 100, 101, 120, 77, 97, 112, 116, 0, 18, 76, 106, 97, 118, 97, 47, 108, 97, 110, 103, 47, 83, 116, 114, 105, 110, 103, 59, 76, 0, 7, 105, 115, 68, 105, 114, 116, 121, 116, 0, 43, 76, 106, 97, 118, 97, 47, 117, 116, 105, 108, 47, 99, 111, 110, 99, 117, 114, 114, 101, 110, 116, 47, 97, 116, 111, 109, 105, 99, 47, 65, 116, 111, 109, 105, 99, 66, 111, 111, 108, 101, 97, 110, 59, 120, 112, 115, 114, 0, 22, 111, 114, 103, 46, 100, 105, 122, 105, 116, 97, 114, 116, 46, 110, 111, 50, 46, 73, 110,... [1.4.200/0]
2022-03-22 07:13:04.999 24743-24743/? W/System.err: at org.h2.mvstore.DataUtils.newIllegalArgumentException(DataUtils.java:1)
2022-03-22 07:13:04.999 24743-24743/? W/System.err: at org.h2.mvstore.type.ObjectDataType.deserialize(ObjectDataType.java:4)
2022-03-22 07:13:04.999 24743-24743/? W/System.err: at org.h2.mvstore.type.ObjectDataType$SerializedObjectType.read(ObjectDataType.java:6)
2022-03-22 07:13:04.999 24743-24743/? W/System.err: at org.h2.mvstore.type.ObjectDataType.read(ObjectDataType.java:6)
2022-03-22 07:13:04.999 24743-24743/? W/System.err: at org.h2.mvstore.type.ObjectDataType.read(ObjectDataType.java:1)
2022-03-22 07:13:04.999 24743-24743/? W/System.err: at org.h2.mvstore.Page$Leaf.readPayLoad(Page.java:3)
2022-03-22 07:13:04.999 24743-24743/? W/System.err: at org.h2.mvstore.Page.read(Page.java:24)
2022-03-22 07:13:04.999 24743-24743/? W/System.err: at org.h2.mvstore.Page.read(Page.java:5)
2022-03-22 07:13:04.999 24743-24743/? W/System.err: at org.h2.mvstore.MVStore.readPage(MVStore.java:4)
2022-03-22 07:13:04.999 24743-24743/? W/System.err: ... 32 more
2022-03-22 07:13:04.999 24743-24743/? W/System.err: Caused by: java.io.InvalidClassException: org.dizitart.no2.Index; local class incompatible: stream classdesc serialVersionUID = 6727331412761083161, local class serialVersionUID = -80130138031701108

Any help / guidance on this is appreciated, as you can understand this has been cause of many sleepless nights for us.

Cheers
HD

[anidotnet]

I'll take a look and let you know

[anidotnet]

You can try the latest release 3.4.4 once and let me know if you are still facing the issue. And please don't forget to attach stacktrace like you did earlier.

[unixguru2k]

Updated to 3.4.4 and re-ran it on emulator

Got this on emulator

2022-03-23 01:14:34.805 25542-25542/messenger.x.chat.bot.messenger.debug W/System.err: java.lang.IllegalStateException: Unable to read the page at position 549755944666 [1.4.200/6]
2022-03-23 01:14:34.806 25542-25542/messenger.x.chat.bot.messenger.debug W/System.err: at org.h2.mvstore.DataUtils.newIllegalStateException(DataUtils.java:950)
2022-03-23 01:14:34.806 25542-25542/messenger.x.chat.bot.messenger.debug W/System.err: at org.h2.mvstore.MVStore.readPage(MVStore.java:2213)
2022-03-23 01:14:34.806 25542-25542/messenger.x.chat.bot.messenger.debug W/System.err: at org.h2.mvstore.MVMap.readPage(MVMap.java:672)
2022-03-23 01:14:34.806 25542-25542/messenger.x.chat.bot.messenger.debug W/System.err: at org.h2.mvstore.Page$NonLeaf.getChildPage(Page.java:1043)
2022-03-23 01:14:34.806 25542-25542/messenger.x.chat.bot.messenger.debug W/System.err: at org.h2.mvstore.Page.get(Page.java:222)
2022-03-23 01:14:34.806 25542-25542/messenger.x.chat.bot.messenger.debug W/System.err: at org.h2.mvstore.MVMap.get(MVMap.java:472)
2022-03-23 01:14:34.806 25542-25542/messenger.x.chat.bot.messenger.debug W/System.err: at org.h2.mvstore.MVMap.get(MVMap.java:459)
2022-03-23 01:14:34.806 25542-25542/messenger.x.chat.bot.messenger.debug W/System.err: at org.h2.mvstore.MVMap.containsKey(MVMap.java:477)
2022-03-23 01:14:34.806 25542-25542/messenger.x.chat.bot.messenger.debug W/System.err: at org.dizitart.no2.store.NitriteMVMap.containsKey(NitriteMVMap.java:49)
2022-03-23 01:14:34.806 25542-25542/messenger.x.chat.bot.messenger.debug W/System.err: at org.dizitart.no2.internals.IndexMetaService.hasIndex(IndexMetaService.java:60)
2022-03-23 01:14:34.806 25542-25542/messenger.x.chat.bot.messenger.debug W/System.err: at org.dizitart.no2.internals.NitriteService.hasIndex(NitriteService.java:93)
2022-03-23 01:14:34.806 25542-25542/messenger.x.chat.bot.messenger.debug W/System.err: at org.dizitart.no2.internals.DefaultNitriteCollection.hasIndex(DefaultNitriteCollection.java:114)
2022-03-23 01:14:34.807 25542-25542/messenger.x.chat.bot.messenger.debug W/System.err: at org.dizitart.no2.objects.DefaultObjectRepository.createIndexes(DefaultObjectRepository.java:317)
2022-03-23 01:14:34.807 25542-25542/messenger.x.chat.bot.messenger.debug W/System.err: at org.dizitart.no2.objects.DefaultObjectRepository.initRepository(DefaultObjectRepository.java:303)
2022-03-23 01:14:34.807 25542-25542/messenger.x.chat.bot.messenger.debug W/System.err: at org.dizitart.no2.objects.DefaultObjectRepository.(DefaultObjectRepository.java:57)
2022-03-23 01:14:34.807 25542-25542/messenger.x.chat.bot.messenger.debug W/System.err: at org.dizitart.no2.objects.RepositoryFactory.open(RepositoryFactory.java:46)
2022-03-23 01:14:34.807 25542-25542/messenger.x.chat.bot.messenger.debug W/System.err: at org.dizitart.no2.Nitrite.getRepository(Nitrite.java:140)
2022-03-23 01:14:34.807 25542-25542/messenger.x.chat.bot.messenger.debug W/System.err: at com.machaao.android.sdk.Machaao.getDB(Machaao.java:2446)
2022-03-23 01:14:34.807 25542-25542/messenger.x.chat.bot.messenger.debug W/System.err: at com.machaao.android.sdk.Machaao.getDB(Machaao.java:1384)
2022-03-23 01:14:34.807 25542-25542/messenger.x.chat.bot.messenger.debug W/System.err: at com.machaao.android.sdk.Machaao.getConfigRepository(Machaao.java:1413)
2022-03-23 01:14:34.807 25542-25542/messenger.x.chat.bot.messenger.debug W/System.err: at com.machaao.android.sdk.Machaao.init(Machaao.java:809)
2022-03-23 01:14:34.807 25542-25542/messenger.x.chat.bot.messenger.debug W/System.err: at com.machaao.android.sdk.Machaao.initAsync(Machaao.java:557)
2022-03-23 01:14:34.807 25542-25542/messenger.x.chat.bot.messenger.debug W/System.err: at messenger.x.chat.bot.messenger.Application.onCreate(Application.java:28)
2022-03-23 01:14:34.807 25542-25542/messenger.x.chat.bot.messenger.debug W/System.err: at android.app.Instrumentation.callApplicationOnCreate(Instrumentation.java:1119)
2022-03-23 01:14:34.807 25542-25542/messenger.x.chat.bot.messenger.debug W/System.err: at android.app.ActivityThread.handleBindApplication(ActivityThread.java:5740)
2022-03-23 01:14:34.807 25542-25542/messenger.x.chat.bot.messenger.debug W/System.err: at android.app.ActivityThread.-wrap1(Unknown Source:0)
2022-03-23 01:14:34.808 25542-25542/messenger.x.chat.bot.messenger.debug W/System.err: at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1656)
2022-03-23 01:14:34.808 25542-25542/messenger.x.chat.bot.messenger.debug W/System.err: at android.os.Handler.dispatchMessage(Handler.java:106)
2022-03-23 01:14:34.810 25542-25542/messenger.x.chat.bot.messenger.debug W/System.err: at android.os.Looper.loop(Looper.java:164)
2022-03-23 01:14:34.810 25542-25542/messenger.x.chat.bot.messenger.debug W/System.err: at android.app.ActivityThread.main(ActivityThread.java:6494)
2022-03-23 01:14:34.810 25542-25542/messenger.x.chat.bot.messenger.debug W/System.err: at java.lang.reflect.Method.invoke(Native Method)
2022-03-23 01:14:34.810 25542-25542/messenger.x.chat.bot.messenger.debug W/System.err: at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:438)
2022-03-23 01:14:34.810 25542-25542/messenger.x.chat.bot.messenger.debug W/System.err: at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:807)
2022-03-23 01:14:34.811 25542-25542/messenger.x.chat.bot.messenger.debug W/System.err: Caused by: java.lang.IllegalArgumentException: Could not deserialize [-84, -19, 0, 5, 115, 114, 0, 53, 111, 114, 103, 46, 100, 105, 122, 105, 116, 97, 114, 116, 46, 110, 111, 50, 46, 105, 110, 116, 101, 114, 110, 97, 108, 115, 46, 73, 110, 100, 101, 120, 77, 101, 116, 97, 83, 101, 114, 118, 105, 99, 101, 36, 73, 110, 100, 101, 120, 77, 101, 116, 97, 19, -115, 41, 124, -29, 79, -77, -36, 2, 0, 3, 76, 0, 5, 105, 110, 100, 101, 120, 116, 0, 24, 76, 111, 114, 103, 47, 100, 105, 122, 105, 116, 97, 114, 116, 47, 110, 111, 50, 47, 73, 110, 100, 101, 120, 59, 76, 0, 8, 105, 110, 100, 101, 120, 77, 97, 112, 116, 0, 18, 76, 106, 97, 118, 97, 47, 108, 97, 110, 103, 47, 83, 116, 114, 105, 110, 103, 59, 76, 0, 7, 105, 115, 68, 105, 114, 116, 121, 116, 0, 43, 76, 106, 97, 118, 97, 47, 117, 116, 105, 108, 47, 99, 111, 110, 99, 117, 114, 114, 101, 110, 116, 47, 97, 116, 111, 109, 105, 99, 47, 65, 116, 111, 109, 105, 99, 66, 111, 111, 108, 101, 97, 110, 59, 120, 112, 115, 114, 0, 22, 111, 114, 103, 46, 100, 105, 122, 105, 116, 97, 114, 116, 46, 110, 111, 50, 46, 73, 110... [1.4.200/0]
2022-03-23 01:14:34.811 25542-25542/messenger.x.chat.bot.messenger.debug W/System.err: at org.h2.mvstore.DataUtils.newIllegalArgumentException(DataUtils.java:924)
2022-03-23 01:14:34.812 25542-25542/messenger.x.chat.bot.messenger.debug W/System.err: at org.h2.mvstore.type.ObjectDataType.deserialize(ObjectDataType.java:376)
2022-03-23 01:14:34.812 25542-25542/messenger.x.chat.bot.messenger.debug W/System.err: at org.h2.mvstore.type.ObjectDataType$SerializedObjectType.read(ObjectDataType.java:1547)
2022-03-23 01:14:34.812 25542-25542/messenger.x.chat.bot.messenger.debug W/System.err: at org.h2.mvstore.type.ObjectDataType.read(ObjectDataType.java:231)
2022-03-23 01:14:34.814 25542-25542/messenger.x.chat.bot.messenger.debug W/System.err: at org.h2.mvstore.type.ObjectDataType.read(ObjectDataType.java:114)
2022-03-23 01:14:34.815 25542-25542/messenger.x.chat.bot.messenger.debug W/System.err: at org.h2.mvstore.Page$Leaf.readPayLoad(Page.java:1511)
2022-03-23 01:14:34.815 25542-25542/messenger.x.chat.bot.messenger.debug W/System.err: at org.h2.mvstore.Page.read(Page.java:607)
2022-03-23 01:14:34.815 25542-25542/messenger.x.chat.bot.messenger.debug W/System.err: at org.h2.mvstore.Page.read(Page.java:239)
2022-03-23 01:14:34.815 25542-25542/messenger.x.chat.bot.messenger.debug W/System.err: at org.h2.mvstore.MVStore.readPage(MVStore.java:2211)
2022-03-23 01:14:34.815 25542-25542/messenger.x.chat.bot.messenger.debug W/System.err: ... 31 more
2022-03-23 01:14:34.817 25542-25542/messenger.x.chat.bot.messenger.debug W/System.err: Caused by: java.io.InvalidClassException: org.dizitart.no2.internals.IndexMetaService$IndexMeta; local class incompatible: stream classdesc serialVersionUID = 1408827874803168220, local class serialVersionUID = 1647971212
2022-03-23 01:14:34.817 25542-25542/messenger.x.chat.bot.messenger.debug W/System.err: at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:606)
2022-03-23 01:14:34.817 25542-25542/messenger.x.chat.bot.messenger.debug W/System.err: at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1626)
2022-03-23 01:14:34.817 25542-25542/messenger.x.chat.bot.messenger.debug W/System.err: at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1521)
2022-03-23 01:14:34.817 25542-25542/messenger.x.chat.bot.messenger.debug W/System.err: at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1777)
2022-03-23 01:14:34.817 25542-25542/messenger.x.chat.bot.messenger.debug W/System.err: at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1354)
2022-03-23 01:14:34.817 25542-25542/messenger.x.chat.bot.messenger.debug W/System.err: at java.io.ObjectInputStream.readObject(ObjectInputStream.java:374)
2022-03-23 01:14:34.818 25542-25542/messenger.x.chat.bot.messenger.debug W/System.err: at org.h2.mvstore.type.ObjectDataType.deserialize(ObjectDataType.java:374)
2022-03-23 01:14:34.818 25542-25542/messenger.x.chat.bot.messenger.debug W/System.err: ... 38 more

[anidotnet]

You need to clear the old data and test. Some internal classes did not have serialVersionUID declared which I fixed in latest version. That's why you are seeing this error - local class incompatible.

As all these internal classes are index related, you can clear the old index and recreate them, which might solve the issue. But I'll suggest to clear old data and test whenever possible.