Restore in-memory index

[link2xt]

It was introduced in #632 and reverted in #643 because of the bug #640

We can restore it once chatmail/core#7005 is solved in the core.

[link2xt]

I am not sure this INDEX=MEMORY is a good idea. It creates a new index for each connection apparently. If we want to move it off the disk then maybe better move it to a tmpfs partition:
https://doc.dovecot.org/2.3/configuration_manual/mail_location/#mail-location-settings-index-files

A new tmpfs partition can be created with a systemd unit, see for example:

$ systemctl cat tmp.mount
# /usr/lib/systemd/system/tmp.mount
#  SPDX-License-Identifier: LGPL-2.1-or-later
#
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.

[Unit]
Description=Temporary Directory /tmp
Documentation=https://systemd.io/TEMPORARY_DIRECTORIES
Documentation=man:file-hierarchy(7)
Documentation=https://systemd.io/API_FILE_SYSTEMS
ConditionPathIsSymbolicLink=!/tmp
DefaultDependencies=no
Conflicts=umount.target
Before=local-fs.target umount.target
After=swap.target

[Mount]
What=tmpfs
Where=/tmp
Type=tmpfs
Options=mode=1777,strictatime,nosuid,nodev,size=50%%,nr_inodes=1m,x-systemd.graceful-option=usrquota

Then it is still in memory, but at least shared. This will however likely become a problem for small server that have disk but not much memory.

[missytake]

Unfortunately, #691 did not actually limit the cache size, it's still growing beyond 500k.

[missytake]

I am not sure this INDEX=MEMORY is a good idea. It creates a new index for each connection apparently. If we want to move it off the disk then maybe better move it to a tmpfs partition: https://doc.dovecot.org/2.3/configuration_manual/mail_location/#mail-location-settings-index-files

A new tmpfs partition can be created with a systemd unit, see for example:

Maybe we can just use https://linuxvox.com/blog/devshm-in-linux/ instead of shipping our own tmpfs. /dev/shm uses at most half of the physical memory of a system; then it reports "file system full", I assume.

I tried to find out how dovecot would handle that - the only info I could find is that dovecot handles it gracefully by acting like INDEX=MEMORY if no space is available - that's what https://doc.dovecot.org/2.3/configuration_manual/quota/quota_fs/#index-files says, but it is talking about the quota plugin driver fs (we use maildir), so I'm not sure if it applies. Probably it does, though.
Then, if /dev/shm is full, it would use RAM, and if that is full, it would start swapping I guess. This will need attention by operators, I assume; but simply restarting the service is a quick remedy, a glance into journalctl -u dovecot should reveal that /dev/shm is full, and then operators can just delete the dovecot index, set INDEX=MEMORY, or move the INDEX to another disk (which seems to be recommended for up-scaled dovecot clusters).

So I think it's kinda okay; we should try it out with a larger instance and a smaller instance first, to see whether we should make it easily configurable in chatmail.ini, as we did with #734 (comment).

[missytake]

It still doesn't help with the fact that dovecot.index and especially dovecot.index.cache files grow larger and larger, and should get pruned every once in a while. We have a lot of throughput per mailbox, and especially the cache files seem to keep around stale data: https://pyte.dev/posts/dovecot-index-cache-issues/

As a part of chatmail-expire, we could delete any dovecot.index.cache files which are larger than 500k, This would mirror what #691 tried to achieve.