Unable to read backup

[Zahrun]

Hi there,

I have accumulated 7 backup files along the years.
With your tool, I am able to decode 6 of them.
Thank you for that !

There is one of them "signal-2019-08-27-14-01-30.backup" which I can’t decode.
My previous backup is in v21 and the following one is in v34, so this particular one should be in any version between these two.

If I use a wrong key, it fails pretty straightforwardly.
I am not sure if it is the right key, but with the key of the previous backup, your tool is trying to do things and finds some things but eventually fails.

The log starts with

*** Starting log: 2025-01-18 12:42:41 ***
signalbackup-tools (signalbackup-tools) source version 20241220.161546 (SQlite: 3.47.0, OpenSSL: OpenSSL 3.3.2 3 Sep 2024)
Please provide passphrase for input file '**/signal-2019-08-27-14-01-30.backup': ***** ***** ***** ***** ***** *****

BACKUPFILE VERSION: 0
BACKUPFILE SIZE: 294991642
COUNTER: 2594777134

Reading backup file: 000.0%...
Reading backup file: 000.0%...


[Warning]: Bad MAC in frame: theirMac: (hex:) ** ** ** ** ** ** ** ** ** **
                              ourMac: (hex:) ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** **
[Error]: Failed to get valid frame from decoded data...
         Encrypted data had failed verification (Bad MAC)
Starting bruteforcing offset to next valid frame... starting after: 32642

Checking offset 10 bytes
Checking offset 20 bytes
Checking offset 30 bytes
Checking offset 40 bytes
Checking offset 50 bytes
Checking offset 60 bytes
Checking offset 70 bytes
Checking offset 80 bytes
Checking offset 90 bytes
Checking offset 100 bytes
Checking offset 110 bytes
Checking offset 120 bytes
Checking offset 130 bytes
Checking offset 140 bytes
Checking offset 150 bytes
Checking offset 160 bytes
Checking offset 170 bytes
Checking offset 180 bytes

GOT GOOD MAC AT OFFSET 184 BYTES!
Now let's try and find out how many frames we skipped to get here....

And finally ends with

Checking if we skipped 96153 frames... nope! :(
Checking if we skipped 96153 frames... 
Checking if we skipped 96154 frames... YEAH! :)
Good frame: 3666602 (StickerFrame)
Frame number: 3666602
        Size: 11
        Type: STICKER
         - length          : 1743079412 (8 bytes)
         - row id          : 18446744073709551615 (8 bytes)
Got frame, breaking

Reading backup file: 615.9%...
[Error]: Failed to read 4 bytes from file to get next frame size... (-1 / 294991642)
[Error]: Unexpectedly hit end of file!
[Warning]: EndFrame was not read: backup is probably incomplete
Database version: 1844960005

The entire log file is 1.9 GiB so I tried to extract some of the relevant information.
Is my backup corrupt? Or is the key wrong? Or is it an issue with the tool?
Thank you again

[bepaald]

Hi!

It's been a long time since I looked at this part of the code, since it's not relevant for modern backups anymore. But I refreshed my memory a little, so here goes:

Is my backup corrupt? Or is the key wrong? Or is it an issue with the tool?

I believe your backup is corrupt. If the key was wrong, the very first (encrypted) frame read will appear corrupt. This is always at a file offset of around 58 bytes (in your case it's 32642 bytes) and a warning message will be printed. You have likely seen this when trying with a known incorrect passphrase.
The tool (I believe) does everything possible to do in an automated way to recover any data still present in the backup file. Looking over the code today, I still think that is the case. There are certain specific types of corruption that the tool will not recover from, but those would require manually manipulating the data (and are astronomically unlikely, for example: all data is actually good but all checksums in the file happen to be corrupt).

There were a few things I thought of when reading your message:

• In the end it has a false positive when it finds a 'valid' StickerFrame. This frame is found after skipping 96154 frames. this number of skipped frames should be impossible after an offset of 184 bytes. Looking at the code though, it should already detect this. Am I correct in assuming there is one (or multiple) messages No valid frame found at maximum frameskip for this offset..., after which the tool returns to Checking offset xx bytes, that were omitted from the part of the log pasted?
• The final frame found is validated, but the validation for sticker frames was not very thorough (a sticker size of 1.6GiB is of course not possible). I have now updated the frame validation for this, so if you were to update and try again, it should not consider this frame valid and keep scanning. Theoretically it may find a true uncorrupt portion of the backup file and recover more data (do not get your hopes up though).
• I was thinking about those first 32642 bytes of correct data. I have just updated the tool to at least be able to dump what is there simply by adding -o [outputdir] (make sure the 'outputdir' is a directory). Though 32642 bytes is very small, at least some messages are likely to be in there when inspecting the SQLite database. I'm assuming, since you have 7 backups, included a newer one, you are not losing data? Or are they not incremental backups?

Let me know if you have any more questions, or if there is anything more you think I can do for you.

Thanks!

[Zahrun]

Hi!

Thank you for looking into the case deeply, and thank you for the detailed answer.

• You are correct, I got No valid frame found at maximum frameskip for this offset 207 times in version 20241220. I also got 1751 times [Error]: Failed to get valid frame from decoded data... Data was verified ok, but does not represent a valid frame... Don't know what happened, but it's bad... :(
  and a ton of [Error]: Unknown wiretype (6). and [Error]: Unknown wiretype (7).
• I updated to version 20250122 and the reading completes with

Checking if we skipped 333567 frames... YEAH! :)
Good frame at offset 7957068. Frame number: 12801993 (Type: StickerFrame)
Frame number: 12801993
        Size: 81
        Type: STICKER
         - row id          : 10517578408941928824 (8 bytes)
         - length          : 3818290993 (70 bytes)
Got frame, breaking
Reading backup file: 1387.8%...
[Error]: Failed to read 4 bytes from file to get next frame size... (-1 / 294991642)
[Error]: Unexpectedly hit end of file!
[Warning]: EndFrame was not read: backup is probably incomplete
Database version: 4294967295

Exporting backup into 'signal-2019-08-27-14-01-30/'
Writing HeaderFrame...
Writing DatabaseVersionFrame...
Writing Attachments...
Writing Avatars...
Writing SharedPrefFrame(s)...
Writing KeyValueFrame(s)...
Writing StickerFrames...
[Error]: STOPPING BEFORE END OF ATTACHMENT!!! (EOF) 
[Error]: Failed to write sticker data
Writing EndFrame...
Writing database...
Saved database to file 'signal-2019-08-27-14-01-30/database.sqlite'
Done!
[Error]: Failed to export backup to 'signal-2019-08-27-14-01-30'

• I get 78 messages in database.sqlite when exporting to a directory with 20250122. I thought the backups were incremental, but it seems the first three aren’t. I probably lost 52 days of chat, it’s not a big deal.

Thanks!

[bepaald]

Well, that does look better, but — and I hate to tell you — it seems I still didn't get the frame validation function correct. The frame found in the end (again a StickerFrame), is quite clearly invalid.

I have updated the validation function again this morning, so I believe if you were to run it again it will not stop at that frame again (though it's hard for me to test). I don't know how long running the tool on that broken backup takes, but I can understand if you see no point in trying again. Especially since (1) I don't think the outcome will be much different in the end and (2) there may very well be yet another false positive (for some frame types, detecting this is much harder (or impossible) than for these StickerFrames).

I'll leave it to you to try again and report back, or just close the issue if you feel you are done with it.

Thanks for the feedback so far!

[Zahrun]

Ok, I decided to build from source this time so that I can get your updates without waiting on nixpkg to update.

This time, with source version 20250128, there seem to be a different issue, the operations stop earlier, but I get the same number of messages in the database.

Checking if we skipped 578097 frames... YEAH! :)
Good frame at offset 8786275. Frame number: 1008830 (Type: AttachmentFrame)
Frame number: 1008830
        Size: 11
        Type: ATTACHMENT
         - row id          : 47 (8 bytes)
         - length          : 1046266350 (8 bytes)
Got frame, breaking
Reading backup file: 361.0%...
[Error]: Failed to read 4 bytes from file to get next frame size... (-1 / 294991642)
[Error]: Unexpectedly hit end of file!
[Warning]: EndFrame was not read: backup is probably incomplete
Database version: 36

Exporting backup into 'signal-2019-08-27-14-01-30b/'
Writing HeaderFrame...
Writing DatabaseVersionFrame...
Writing Attachments...
[Error]: STOPPING BEFORE END OF ATTACHMENT!!! (EOF) 
[Error]: Failed to retrieve attachment data for attachment (rowid: 47 uniqueid: -1)
Writing Avatars...
Writing SharedPrefFrame(s)...
Writing KeyValueFrame(s)...
Writing StickerFrames...
Writing EndFrame...
Writing database...
Saved database to file 'signal-2019-08-27-14-01-30b/database.sqlite'
Done!
[Error]: Failed to export backup to 'signal-2019-08-27-14-01-30b'

[bepaald]

Thanks again.

This time, with source version 20250128, there seem to be a different issue, the operations stop earlier,...

Looking at the offset compared to your previous message (Good frame at offset 8786275 vs 7957068), it actually stops later. Which was expected since the StickerFrame found previously does not pass validation anymore.

The frame it now finds, has basically all the right values to be a real frame, but I thought of one more way to detect the validity of the frame in this case: the length of the attachment is more than there are bytes left in the file. So I've updated the frame validation routines again: now all attachment-type frames will also check their size compared to the file offset and total size.

but I get the same number of messages in the database.

Well, yes, unless there is actually some valid (non-corrupted) data in the file (and it is found), the number of messages is not going to get higher. You could try again with these last changes applied, but I'm not very hopeful it will help you recover more data (though depending on how much trouble it is building and running the tool, it can't hurt).

Either way your feedback has certainly helped me make the validation functions more precise, I am thankful for that.

[Zahrun]

Looking at the offset compared to your previous message (Good frame at offset 8786275 vs 7957068), it actually stops later. Which was expected since the StickerFrame found previously does not pass validation anymore.

I assumed it was less due to the fact that the processing happens much faster and in the output folder I get much fewer files "KeyValue" and "SharedPreference". I got 9 files where in 20241220 I was getting 42 files. The log is 490 MiB instead of 1.9 GiB.

So I've updated the frame validation routines again: now all attachment-type frames will also check their size compared to the file offset and total size.

Ok now in 20250128.165407 I get a better ending message, 65 files in the output folder, a log of 6.6 GiB, and still 78 messages.

[Error]: Failed to read next frame (539585408 bytes at filepos 294912118)
Starting bruteforcing offset to next valid frame... starting after: 294912118
Checking offset 54570 bytes

GOT GOOD MAC AT OFFSET 54576 BYTES!
Now let's try and find out how many frames we skipped to get here....
Checking if we skipped 1763 frames... nope! :(
[Error]: Unknown wiretype (6).
Checking if we skipped 2144 frames... nope! :(
[Error]: Unknown wiretype (6).
Checking if we skipped 5458 frames... nope! :(

No valid frame found at maximum frameskip for this offset...
Checking offset 79520 bytes
[Error]: Failed to read 4 bytes from file to get next frame size... (-1 / 294991642)
[Error]: Failed to read 4 bytes from file to get next frame size... (-1 / 294991642)
[Error]: Failed to read 4 bytes from file to get next frame size... (-1 / 294991642)
[Warning]: EndFrame was not read: backup is probably incomplete
Database version: 94

Exporting backup into 'signal-2019-08-27-14-01-30c/'
Writing HeaderFrame...
Writing DatabaseVersionFrame...
Writing Attachments...
Writing Avatars...
Writing SharedPrefFrame(s)...
Writing KeyValueFrame(s)...
Writing StickerFrames...
Writing EndFrame...
Writing database...
Saved database to file 'signal-2019-08-27-14-01-30c/database.sqlite'
Done!

[bepaald]

It looks like it scanned through the entire file without finding any valid, decryptable data (and no false positives). I feel fairly confident that you got everything out of that file that was possible to get.

Thank you again for the feedback, you've helped improve this tool. And sorry for the loss of those few messages. I'm going to close this issue, as I think there is nothing left to be done here, but if you have anything to add or ask, of course feel free.

Thanks again!

[Zahrun]

Thank you so much for your diligent support. I am glad that we could get to the bottom of it and that it could help improve the tool.

Best wishes.