[FEATURE] Implement an SQL-based backend (to avoid ObjectBox)

[hatch01]

What do you want implemented?

Now, the package uses objectbox which is blacklisted by fdroid, so I can't deploy my app to fdroid.

2024-09-05 18:59:54,513 DEBUG: Problem: found class 'io/objectbox/converter/IntegerLongMapConverter'
2024-09-05 18:59:54,521 DEBUG: Problem: found class 'io/objectbox/exception/UniqueViolationException'
2024-09-05 18:59:54,533 DEBUG: Problem: found class 'io/objectbox/converter/PropertyConverter'
2024-09-05 18:59:54,535 DEBUG: Problem: found class 'io/objectbox/exception/DbExceptionListener'
2024-09-05 18:59:54,542 DEBUG: Problem: found class 'io/objectbox/exception/NumericOverflowException'
2024-09-05 18:59:54,558 DEBUG: Problem: found class 'io/objectbox/tree/LeafNode'
2024-09-05 18:59:54,567 DEBUG: Problem: found class 'io/objectbox/exception/DbException'
2024-09-05 18:59:54,579 DEBUG: Problem: found class 'io/objectbox/exception/DbExceptionListenerJni'
2024-09-05 18:59:54,607 DEBUG: Problem: found class 'io/objectbox/exception/ConstraintViolationException'
2024-09-05 18:59:54,616 DEBUG: Problem: found class 'io/objectbox/converter/LongFlexMapConverter'
2024-09-05 18:59:54,621 DEBUG: Problem: found class 'io/objectbox/exception/NonUniqueResultException'
2024-09-05 18:59:54,632 DEBUG: Problem: found class 'io/objectbox/converter/StringFlexMapConverter'
2024-09-05 18:59:54,634 DEBUG: Problem: found class 'io/objectbox/converter/StringLongMapConverter'
2024-09-05 18:59:54,644 DEBUG: Problem: found class 'io/objectbox/exception/FileCorruptException'
2024-09-05 18:59:54,650 DEBUG: Problem: found class 'io/objectbox/exception/DbMaxDataSizeExceededException'
2024-09-05 18:59:54,673 DEBUG: Problem: found class 'io/objectbox/exception/PagesCorruptException'
2024-09-05 18:59:54,685 DEBUG: Problem: found class 'io/objectbox/Transaction'
2024-09-05 18:59:54,687 DEBUG: Problem: found class 'io/objectbox/exception/DbFullException'
2024-09-05 18:59:54,698 DEBUG: Problem: found class 'io/objectbox/converter/NullToEmptyStringConverter'
2024-09-05 18:59:54,701 DEBUG: Problem: found class 'io/objectbox/annotation/Entity'
2024-09-05 18:59:54,714 DEBUG: Problem: found class 'io/objectbox/converter/LongLongMapConverter'
2024-09-05 18:59:54,718 DEBUG: Problem: found class 'io/objectbox/exception/DbDetachedException'
2024-09-05 18:59:54,718 DEBUG: Problem: found class 'io/objectbox/exception/DbShutdownException'
2024-09-05 18:59:54,728 DEBUG: Problem: found class 'io/objectbox/exception/DbMaxReadersExceededException'
2024-09-05 18:59:54,733 DEBUG: Problem: found class 'io/objectbox/relation/ToOne'
2024-09-05 18:59:54,744 DEBUG: Problem: found class 'io/objectbox/converter/FlexObjectConverter'
2024-09-05 18:59:54,753 DEBUG: Problem: found class 'io/objectbox/exception/a'
2024-09-05 18:59:54,757 DEBUG: Problem: found class 'io/objectbox/BoxStore'
2024-09-05 18:59:54,760 DEBUG: Problem: found class 'io/objectbox/converter/StringMapConverter'
2024-09-05 18:59:54,764 DEBUG: Problem: found class 'io/objectbox/exception/DbSchemaException'
2024-09-05 18:59:54,795 DEBUG: Problem: found class 'io/objectbox/relation/ToMany'
2024-09-05 18:59:54,804 DEBUG: Problem: found class 'io/objectbox/converter/IntegerFlexMapConverter'
2024-09-05 18:59:54,816 CRITICAL: Found 32 problems in build/fr.onyx.lyon1/apps/onyx/build/app/outputs/flutter-apk/app-arm64-v8a-release.apk

For curiosity, why did you moved to objectbox ?

What other alternatives are available?

Juste using a fork or an alternative could also be a solution, but it seems to be a lot of work to maintain a fork with a different database.

Can you provide any other information?

No response

Severity

Obtrusive: No workarounds are available, and this is essential to me

[JaffaKetchup]

Hi @hatch01,

ObjectBox was chosen as it is currently one of the better options available for Dart that provided the tools needed - namely about multi-Isolate access. One point of experimentation for me is to see whether using Isolates for database operations actually makes a difference.
I use Isolates, not because I want simulataneous access to the database necessarily, but because I want to attempt to avoid overheads in copying to/from isolates frequently. I think this may make sense for the bulk downloading stuff, but possibly not the main stuff.
In order to understand this better, there's no option but to test and benchmark myself, which I don't have time for at the moment.

Isar was moved away from as it had too many stability issues - of course, now it turns out ObjectBox has some stability on iOS as well. Hive isn't suitable, and also seems undermaintained alongside Isar by the same maintainer. The IO filesystem from <v6 was too slow and now wouldn't handle the relations required. And performance is also relatively important.

The potentially plausible alternatives are Floor (which is pretty much just a layer over SQL libraries), or using an SQL library directly (probably sqllite3), both of which require manual SQL statements. I'm not new to SQL, and I could probably do it, but I'm certainly not an expert in doing things efficently or the more advanced parts etc.

Luckily, I did it properly (hopefully ;)) when I rewrote FMTC, so it should not be possible just to write a new database backend for a new database library without messing around with all the other code again - much easier. Especially if I can figure the Isolates thing out.

You may be able to write a backend yourself. You can use the existing one as a base, but I believe all the necessary components are exported under the special import.

At the moment, I don't have massive amounts of time to invest into FMTC, especially for things that won't necessarily bring new features or fixes. If I get time, which hopefully won't be too long, I will look into making another backend.

---

It's worth mentioning that the issues with f-droid and ObjectBox licensing are fairly complex. I'm not a license lawyer by any means, but it seems f-froid imposes some heavy restrictions to keep things 100% open source entirely, even beyond build time. It's a valid approach of course, and ObjectBox's licensing situation is somewhat confusing at the least. AFAIK the build time dependencies are open source, but the runtime native dependency is not. Again, not a lawyer, and both f-droid and ObjectBox have perfectly valid approaches in their own ways.

objectbox/objectbox-java#560 (comment)

[hatch01]

Hi,
Thank you for your very detailed answer.
While opening the issue, I was realistic about the unreasonable amount of work I was asking for and that you did not choose objectbox randomly.
I'll move to flutter_map_cache temporarily to still be available on f-droid and if I get time to develop on this package, I will look at adding multiple storage backend.

[hatch01]

Do you want me to close the issue, or may it stay opened for a hypothetical future development?

[JaffaKetchup]

It's not a totally unreasonable amount of work, I was planning that it may need to be done, and there's some other issues with ObjectBox as well. I am just unfortunately a little short of time at the moment.

If flutter_map_cache meets your needs (you don't need bulk downloading or advanced store management), then it's a great option as well!

I'll note this discussion on the documentation.

I'll leave it open but change the title.

[MatKershaw3708]

I am having issues relating to objectbox, this occurs on app startup and if freezes on the splash screen:

[ERROR:flutter/runtime/dart_vm_initializer.cc(41)] Unhandled Exception: Bad state: failed to create store: Cannot open store: another store is still open using the same path: "/data/data/com.myappname.mobile/app_flutter/fmtc

I've obscured the app name for privacy. Let me know if I need to open this as a new issue.

[JaffaKetchup]

Hi @MatKershaw3708,
Please open another issue for this, and let me know if it happens in release and/or debug and whether you've just done a hot reload if in debug. Thanks!

[Gaurav-CareMonitor]

@JaffaKetchup Have you thought of moving to https://pub.dev/packages/drift
Supports web and automatically generates typesafe queries

[lume-code]

One of downsides of Objectbox here is that in order for it to work on macOS, the sandbox should be disabled and as far as I know apps with disabled sandbox are not allowed into App Store. Using ObjectBox eliminated macOS and web at the same time.

[JaffaKetchup]

FMTC works on macOS without being unsandboxed @manlyman29. Please see #174.