Shutting off machine at the wrong time results in "The database disk image is malformed" with the UI being stuck during with no useful indication anything is wrong

[ell1e]

(based on issue deltachat/deltachat-desktop#4842 )

Sometimes, DeltaChat Desktop no longer syncs on one of my devices. This seems to be potentially a core bug. The UI sometimes will say "Updating..." for a while, but the messages shown in the main contact list never update and they're stuck roughly 10 days in the past, and no new messages are showing. Any individual chat won't load, it just say "Select a chat or create a new chat" and nothing ever happens.

Here's how the main contact list looks like while stuck:

During investigations, I obtained a --devmode terminal output log file which shows the following error: 7.2s �[1;33m[w]�[0m�[0;37mcore/event�[0m: 2 src/scheduler.rs:731: Failed fetch_idle: fetch_move_delete: fetch_new_messages: database disk image is malformed: Error code 11: The database disk image is malformed

• Operating System (Linux/Mac/Windows/iOS/Android): Linux
• Core Version: v1.156.2, v1.159.4
• Client Version: 1.54.2, 1.58.2

Expected behavior

Shutting down the device at an unfortunate time won't leave Delta Chat unable to sync after the device starts up again.

Actual behavior

Shutting down the device at an unfortunate time apparently will leave Delta Chat unable to sync after the device starts up again.

Steps to reproduce the problem

1. Use Delta Chat
2. Hard shut off your device with power loss at just the wrong moment
3. I use btrfs. This means any sort of checksum failure due to filesystem corruption should lead to an I/O error on open()/fopen() and not allow any reading in the file, and I think there should also be a dmesg message. Since that didn't happen, I don't think a filesystem corruption is the likely cause here (although I did have that on the PinePhone in the past, likely due to how unreliable SD cards are, so I thought I would point that out to be completely transparent).

Screenshots

see above

Logs

see above

[iequidoo]

This doesn't look as a Core bug, but rather as an SQLite bug (but unlikely) or some problem with sync requests implementation by your device, e.g. they don't work as write barriers (https://www.sqlite.org/howtocorrupt.html mentions this along with other possible reasons, you can check them). Btrfs mustn't necessarily detect such corruptions by checksumming because AFAIU checksums are calculated separately for allocated fs blocks, but if blocks are written in the wrong order by your device, that won't help. Which device do you use?

[ell1e]

The PinePhone uses an SD Card, so I assume "Disk drives that do not honor sync requests" is potentially true. But the contents shouldn't be corrupted since BTRFS would catch that (it did in the past, so that's not theoretical).

[iequidoo]

It's unlikely that Btrfs checksums can protect from reordered writes because they are calculated for blocks independently. You would rather get some fs metadata mismatch, but if just two data (not metadata) blocks are written in the wrong order, i'd not expect the fs to detect anything unless you use an fs with full data journaling (e.g. ext4 has such a mode, Btrfs doesn't). Do you have Btrfs logs of the previous corruption? Which kind of file/software did that happen to, SQLite or smth else?

[ell1e]

Oh it can happen to any large file that you copy on there really, it's probably just SD cards occasionally bit flipping and being SD cards. I wouldn't recommend using SD cards with anything but a filesystem with checksums. It leads to a dmesg message pointing out the checksum error and the open() fails with an EIO or some similar errno value. If you write like, 50GiB of files then the chance of one of them having a bit flip isn't that small on an SD card, I tried multiple card vendors too.

[iequidoo]

So this time it's definitely not a bit flipping. I'd suggest you try another fs, say, ext4 in the default data=ordered mode and see what happens (i'd expect the same corruption). Btw, how often does this reproduce? Then you can mount the fs in the data=journal mode and make more tests. https://serverfault.com/a/914466 explains how data journaling helps to maintain data consistency w/o write barriers working/enabled. But you should make sure the journal is big enough, 102400 blocks is the maximum size apparently.

At least some experiments may help to understand how to use Delta Chat on such devices or falsify assumtions about possible reasons of the problem.

[ell1e]

It only really reproduces every 2-3 weeks or so, the Delta Chat getting stuck event, so it's not very often sadly. I use this device a lot so I can't easily switch out the filesystem. I had ext4 at some point long in the past, but I didn't use Delta Chat yet around that time.

[iequidoo]

I guess, you switched to Btrfs because of ext4 lacking data checksumming? Then Btrfs is better but also not ideal it seems. What you can do is to somehow script Btrfs auto-snapshotting (with rotation of old snapshots) and in case of a hard shutdown just roll back to some sufficiently old snapshot so that all data is already synced by your SD card there. You'll lose some amount of recently fsync()ed data, but anyway your SD card apparently doesn't guarantee the opposite and at least you won't get such a silent data corruption (it's actually silent because well, SQLite detects it, but other software doesn't necessarily). Still i'm not sure that will guarantee fs metadata consistency because if Btrfs moves around some metadata at the moment of a power loss, it may get corrupted, but at least not silently and i guess Btrfs should avoid that (i'm not familiar with the Btrfs source code, but usually doing so is a bad idea).

[ell1e]

I guess, you switched to Btrfs because of ext4 lacking data checksumming

Correct! My guess would be btrfs likely avoids every kind of silent corruption, other than perhaps actually guaranteeing an fsync() goes through if the SD card doesn't reliably report it, which I doubt that it does.