Feature Request: Optional DB storage for user/chan state (SQLite first; MariaDB/PostgreSQL next; Redis optional)

[DiamantTh]

Hey,

Add an optional storage backend so Eggdrop can load/save user & channel state using a database.
Flatfiles stay default. Start with SQLite (embedded, no external service). Later add MariaDB/PostgreSQL. Redis is optional (cache only).

Why:

Reliable saves (safe, all-or-nothing).

Query/backup/audit without parsing flatfiles.

Low risk: wrap current load/save behind a small provider; no TCL command changes.

MVP:

Providers: flatfile (existing, default) + sqlite.

Config switch, one-shot migration from flatfiles, easy rollback via config.

No change to TCL binds/commands for callers.

Details and a Tcl example config are attached in a separate Markdown file.

[crazycatdevs]

I see some potential troubles when using several eggdrops within the same "instance":
With sqlite, you can do a base per eggdrop (based on botnick), ok. But using MariaDB/pgsql, not sure the user has the hability to create several bases. So using table prefixes ? This makes a trouble in sqlite where only one connection can write a base (no concomitents write connections).
So it means you'll have 2 ways of work depending on the database type...

[DiamantTh]

Morning,

thanks for your input! From what I understand, your main concern is that not everyone has unlimited access to create multiple databases, especially with MariaDB or PostgreSQL setups. So, using one shared database with table prefixes would be the preferred way there.

For SQLite, on the other hand, it makes more sense to have separate database files per bot to avoid concurrency issues, since SQLite doesn't handle simultaneous writes well.

I believe I understood your point correctly that there are basically two approaches depending on the database backend and typical hosting restrictions.

Personally, I would consider both options depending on the use case:

For self-hosted setups with full control, using a classic DB (MariaDB/PostgreSQL) with a shared database and table prefixes makes the most sense due to better performance and concurrency.

For smaller or local deployments where ease of setup matters, SQLite3 or similar file-based databases are great, using separate DB files per bot to avoid locking issues.

These thoughts are just some ideas to fuel discussion or provide a starting point. Would be great to hear how you envision handling this within the project or if there are any best practices planned.

Thanks again!

[vanosg]

Hi @DiamantTh ,

I read this when you submitted, but am only getting around to responding now. I'd be curious what 'channel state' and 'user state' means in this context. I do like the idea of implementing database support, but if you're talking about channel and userfiles, the compatibility issues would be significant and require additional user setup (how do you share userfiles between a database bot and a non-database bot, for example)

That said, I want to be clear that I do like the idea of making database support easier/more intuitive because so many scripts use or could benefit from it- but am unsure if this treads too close to something like tclsqlite that already exists. If a database capability were built, it would be an optional, not required, service requirement to run Eggdrop, and thus we can't guarantee every Eggdrop would have it. That means that anything requiring a database could not live in the core code (such as userfiles or channelfiles), it would be an add-on capability.

But again, I'm curious to learn more about what specific use cases or functionality you have in mind and that could help shape the discussion. Thanks for starting the conversation

[DiamantTh]

Hey @vanosg,

I was wondering if something like this would even be possible or make sense from the project’s perspective.

By “channel/user state” I mainly meant the .chan and .user files, but more conceptually — how this data could be organized or extended, not to replace the current file system.

If SQL support is compiled in, could it potentially serve as an optional storage backend for these or related data types?

The idea wouldn’t be to make databases mandatory, but to let Eggdrop take advantage of SQL when available — for example, for user, channel, or script data that could benefit from a more structured layout.

Would such an approach be technically feasible or align with the project’s goals, or do you think it would add too much complexity for too little benefit?

I could imagine it working, but I’m curious how the developers view this idea overall.

Greetings from Upper Franconia

[vanosg]

You're spot on, extending something is a great use, but the complexity comes with how to use incorporate two different ways to run the same type of process, userfile sharing, etc and how to handle that. Not impossible, but definitely "hard", probably more complexity than our tiny volunteer staff can handle. BUT- are there other useful implementations? Extending is good, but duplicating is hard.

So I think in short- redoing the userfile system, while undisputedly a more robust solution, is probably not in the cards. But is there some other use that might make sense as an extension/module?

[michaelortmann]

eggdrop support binds like CHANSET (triggered when a channel setting is set via the partyline). A module or tcl could be written and bind to user and chan adding/removing/changing events and share them with a database. It could also propagate database changes to the bot. This last part is easier for we should have all module/tcl functions needed already. But as far as i know, eggdrop currently does not have event binds for user adding/removing/change or channel adding/removing. If we added all those (missing?) binds, a module / script should be possible to transparently share user and chan data between eggdrop and a database.