What do you think about using pocketbase as a database backend?

[arrkiin]

Hi everyone,

before I stumbled over emissary I also thought about building a simple self-contained golang project for my own personal usage. During that time I also thought about reducing the complexity. So I also come over htmx (https://htmx.org/), golang templates, templ (https://templ.guide/) as an interessting template option and pocketbase (https://pocketbase.io/) as a simple but powerful database backend solution.

What do you think about using pocketbase as a database backend to reduce the external dependency to zero and create a completely self-contained executable with minimal configuration efforts. It is usable as a framework (https://pocketbase.io/docs/use-as-framework/) that works on top of the echo web framework and also provides direct access to it.

[benpate]

Great suggestion, Jens. Emissary is built to work across multiple databases.(Nearly) all access goes through a common adapter -- at github.com/benpate/data. So, if we make a PocketBase adapter for this interface, we'd be 80% of the way there.

There are two exceptions:

1. There are a few custom queries in the /queries folder that will need to be refactored. Some of them can probably be rolled into the abstract data package, but others are custom and will just have to be rewritten for PocketBase

2. More difficult will be primary keys for all of the tables. Mongo uses a custom 24-byte ObjectID for all its primary keys. We'll need to work out how to store/convert these into PocketBase.

[benpate]

I do want to add that it's important to still support MongoDB, in addition to PocketBase, or any other db we might add in the future.

One important goal for Emissary is for it to be really easy to host at both a small and large scale. Embedded PocketBase will help individuals to get it up and running, and MongoDB is important for large hosts to host Emissary at scale.

So, I'm seeing this as a default embedded option that could be changed by the administrators. Does that fit with your idea, too?

[arrkiin]

I really struggled a bit in how I could reply to your sentences. I think there are really many different goals that could drive project decissions and will create much impact on the effort that must be taken. The more flexibility you provide, the more effort you will have to provide and support it.

As a solo dev I really want to reduce my own effort to build things and want to benefit from great project out there, that will support my own goal as much as possible.

As I came over pocketbase I appreciated, that it was build with golang and can be integrated into a project, has an own integrated admin UI, and an authentication and user/role based permission system, that is flexible enough to build a good application backend.

If we talk about scaling, there are many possibilites to scale an application. There is also the question what should scale and where is the possible bottleneck of the application.

Even if emissary will run at scale, how big will one emissary domain/instance be? Is the database as big as you think? Isn't it possible that every domain is an encapsulated executable with a separate database? Probably these instances could be seperate containers.

Regarding the ObjectID from MongoDb, do you think it is possible to replace the ID of emissary to a string and do a conversion from ObjectID to string and vice-versa within the "data" implementation of the corresponding database adapter? As I read through the Pocketbase documentation they also have a special specification for the record id. They use a fixed 15-char long string.

[arrkiin]

By the way, I don't have any experience in building a big scalable application framework. So I cannot really help with that kind of questions. My main focus is in building company and personal websites and even internal company portals as quick, flexible and secure as possible. In addition, they must be highly integrated in existing workflows and software applications. So my point of view is probably a bit different.

[benpate]

Hey @arrkiin - It's been a busy couple of weeks for me, so I apologize for being slow to getting back to you. I'm glad that we're coming at this from two different points of view. It will be a strength when we try to solve this problem like the blind men and the elephant.

So yes, flexibility is expensive. I have tried to follow Clean Architecture as realistically as possible, which (should?) give us the potential for a second database back end in the future. So long as we have a consistent API, this works pretty well; I'm already doing it with file/media storage, letting admins choose between local files, S3 storage, and GitHub (for read-only files). And we have the ability to add other back ends like HTTP, Azure, and Dropbox whenever they become useful. Yes, databases are more complicated, but the principle should still work.

My goal is for Emissary to be easy for an technical person to self-host an individual site (I'm hosting some tester pages on Dreamhost) and for a hosting provider to host thousands (or hundreds of thousands) of databases at scale. One site may serve a single person, a small group like a team or family, or an enormous community of a million or more users. I recognize this is a big range to support, but it's more doable than most people think, and it's worth the effort to try. In the end, sensible defaults will make it possible for amateurs bootstrap, and sane extension points will make the pros happy, too.

About Pocketbase

I've done a little digging into this, and there is one big caveat to using Pocketbase. Pocketbase seems like a table-based SQL database, not a "document" or No-SQL database. That seems like a big difference that might be hard to bridge. Because of the nature of the ActivityPub, Emissary is forced to work with a lot of complex data structures. It would be possible to map all of that into normalized SQL tables, but it would be an enormous job with little payback.

I'm going to dig into https://github.com/gostor/awesome-go-storage#database to see if there are any embeddable, No-SQL databases for Go. Are you familiar with any that I should start with?

[benpate]

What about this? https://github.com/ostafen/clover