Corrupt files and folder when gocryptfs cipherdir is synchronised via Syncthing

[savchenko]

I have started observing this during the last 2 weeks or so, albeit on a smaller scale.

In previous cases corruption was mostly affecting git indexes, but here damage is more extensive:

$ gocryptfs -fsck gocrypt/
Password: 
Decrypting master key
OpenDir ".": invalid entry "Q42GWQmqmH5GvujP8DOrdg": bad message
fsck: corrupt entry in dir "": "Q42GWQmqmH5GvujP8DOrdg"
OpenDir "mhzofRHRP2shWMMtzv6ZEA": could not read gocryptfs.diriv: no such file or directory
fsck: error opening dir "xxxxxxxxx/xxxxxxxxxx/xx/xxxxxxxx/xxxx/xxxxxxxx": 5=input/output error
OpenDir "CmE5cc0HwXERKp3e6lO3gA": invalid entry "BS1jRx94o2SPZE3VVoKduw": bad message
fsck: corrupt entry in dir "xxxx/xxxx/xxx": "BS1jRx94o2SPZE3VVoKduw"
OpenDir "TNYugAMPIEi73eD6AlHZ3A": invalid entry "ErCyzALmrA_6iMQ2je18iQ": bad message
fsck: corrupt entry in dir "xxxx/xxxx/xxx/xxxxxx": "ErCyzALmrA_6iMQ2je18iQ"
OpenDir "YbtQ8679xCEYi324Okt7Bw": invalid entry "Q7lbJVgsU7EQJollS_EUng": bad message
fsck: corrupt entry in dir "xxxx/xxxx/xxx/xxxxxx/xxxxx": "Q7lbJVgsU7EQJollS_EUng"
OpenDir "8eOpARKnAQAbCBLVaVIBgg": invalid entry "r3UIaZYYMDQQjIOfM4WpJQ": bad message
OpenDir "8eOpARKnAQAbCBLVaVIBgg": invalid entry "XHaLuVAw3rTZ0A72ErwQDw": bad message
fsck: corrupt entry in dir "xxxx/xxxx/xxxxxx": "r3UIaZYYMDQQjIOfM4WpJQ"
fsck: corrupt entry in dir "xxxx/xxxx/xxxxxx": "XHaLuVAw3rTZ0A72ErwQDw"
OpenDir "EOiNnPbRmZLFlDZUy1hzRw": invalid entry "J9WwFnSWdFMcRtv88muvpg": bad message
fsck: corrupt entry in dir "xxxx/xxxx/xxxxxx/xxxxxxxxxxx": "J9WwFnSWdFMcRtv88muvpg"
OpenDir "xCcc7WlHYQ-CC31Q-T5yIw": invalid entry "LKwyeRtRF_0Bd-OWgPEAnA": bad message
fsck: corrupt entry in dir "xxxx/xxxx/xxxxxx/xxxxxxxxxxx/xxxx": "LKwyeRtRF_0Bd-OWgPEAnA"
OpenDir "mLOdyr1FCyOnfNa2332zIQ": invalid entry "nuPcy1D81IzB3sKpQiGSFQ": bad message
fsck: corrupt entry in dir "xxxx/xxxx/xxxxxxxx": "nuPcy1D81IzB3sKpQiGSFQ"
OpenDir "h5wPJK5dw21t-8bQLhfkuw": invalid entry "360a5xjQI5NYzUrCQzyb-A": bad message
fsck: corrupt entry in dir "xxxx/xxxx/xxxxxxxx/xxxxxxxx": "360a5xjQI5NYzUrCQzyb-A"
OpenDir "n4buocgPjyZ8eRSeAqCk7g": invalid entry "7Iifijp3w-TGxvt57TVMFw": bad message
fsck: corrupt entry in dir "xxxx/xxxx/xxxxxxxx/xxxxxxxxxxx": "7Iifijp3w-TGxvt57TVMFw"
OpenDir "cOo3UWTinz-Nrzmnojz2Og": invalid entry "Wo87J3NGSRa_D7KfTiF-0Q": bad message
fsck: corrupt entry in dir "xxxx/xxxx/xxxxxxxx/xxxxxxxxx": "Wo87J3NGSRa_D7KfTiF-0Q"
OpenDir "n15XaS9mDqLXjQXYUtkdUw": invalid entry "3cYjofmRbfOomOT_oZvRug": bad message
fsck: corrupt entry in dir "xxxx/xxxx/xxxxxxxxx": "3cYjofmRbfOomOT_oZvRug"
OpenDir "vPa5sYKAdS1HNODvxUnGcw": invalid entry "fNOALh19-Gk_Nfq3f9q5kQ": bad message
fsck: corrupt entry in dir "xxxx/xxxx/xxxxxxxxx/BSD": "fNOALh19-Gk_Nfq3f9q5kQ"
OpenDir "Re1tpEUGMO9Rh2aW8LI_yg": invalid entry "ww0XqKavBZqE3xoTXUPJLA": bad message
fsck: corrupt entry in dir "xxxx/xxxx/xxxxxxxxxxxx": "ww0XqKavBZqE3xoTXUPJLA"
OpenDir "xWrSDaS9XCmZxVin_3UkVA": invalid entry "4NtZBOl8UevnjabTYHT5XA": bad message
fsck: corrupt entry in dir "xxxx/xxxx/xxxxxxxxxxxx/xxxxxx": "4NtZBOl8UevnjabTYHT5XA"
OpenDir "dUr7Tfadq4vJXBetXhrAGw": invalid entry "QsBR3hImqbjav3TYaR7YxQ": bad message
fsck: corrupt entry in dir "xxxx/xxxx/xxxxxxxxxxxx": "QsBR3hImqbjav3TYaR7YxQ"
OpenDir "G0PrT6uQD760thIUu_PZfg": invalid entry "Ie_9LDDFJMkoOv2FhKgueg": bad message
fsck: corrupt entry in dir "xxxx/xxxx/xxxxx": "Ie_9LDDFJMkoOv2FhKgueg"
fsck summary: 19 corrupt files, 0 files skipped

Folder that shows 5=input/output error is displayed in clear-text as empty.

Is there anything I can do to help with investigating this issue? Or maybe there is some underlying incompatibility of GoCrypt share being synchronised bi-directionally?

Thanks.

I have reached the same issue. I have 3 devices (server, PC and android) and I am syncing every device with all the others.

I do not see "corrupt entry errors" but I have OpenDir ".": invalid entry ".stfolder": illegal base64 data at input byte 0
`

Because of this I have moved to https://cryptomator.org/ for now. It seems to work well with syncthing so far. I would prefer gocryptfs because it has fully open-sourced android app, but until this issue is solved, it is not possible.

[rfjakob]

I see three different issues here:

1. OpenDir "mhzofRHRP2shWMMtzv6ZEA": could not read gocryptfs.diriv: no such file or directory

Looks like syncthing forgot to copy the gocryptfs.diriv file (what!?)

2. OpenDir "CmE5cc0HwXERKp3e6lO3gA": invalid entry "BS1jRx94o2SPZE3VVoKduw": bad message

The file names in the folder do not match the gocryptfs.diriv in that folder (huh!?)

3. OpenDir ".": invalid entry ".stfolder": illegal base64 data at input byte 0

Syncthing created a file .stfolder, and gocryptfs cannot decrypt it (obviously). I suggest you put the gocryptfs encrypted dir in a SUBFOLDER of the folder you sync with syncthing.

[savchenko]

Syncthing created a file .stfolder, and gocryptfs cannot decrypt it (obviously). I suggest you put the gocryptfs encrypted dir in a SUBFOLDER of the folder you sync with syncthing.

I will double-check the previous points, but can say now that gocryptfs-encrypted folder is already in a subfolder of the folder synced.

Please let me know if there is anything else that might help you to debug this.

[rfjakob]

@savchenko i only saw the .stlfolder in the messages from @peterflock

[rfjakob]

@savchenko Do you see the same corruption on all synchronised copies or does it look different?

[rfjakob]

Question to all syncthing users here: Do you use the -sharedstorage flag (you should!)?

I started syncthing three times like this

cd /var/tmp
mkdir a b c
syncthing -home=a &
syncthing -home=b &
syncthing -home=c &

and synced a folder between all three, so that I have three copies on disk, and mounted the three copies three times (without using `-sharedstorage). And after some trying I did manage to reprodruce git corruption:

$ git fsck
Checking object directories: 100% (256/256), done.
error: 60ac6e68fa5f74a0ae8cf39f72237e7b2b43cbf0: invalid sha1 pointer in cache-tree
broken link from    tree 979c573280be7d111e2874c0d7427df87171d20a
              to    tree 85a49a1f0b6b87b639763f280b5c7982cec06a81
...

[rfjakob]

Todos for go-fuse to make things more robust even without -sharedstorage:

1. Clear parents map if out.Nlink == 1? ---> https://github.com/hanwen/go-fuse/blob/0f728ba15b38579efefc3dc47821882ca18ffea7/fs/bridge.go#L210

2. Select last know-good parent instead of random? ---> https://github.com/hanwen/go-fuse/blob/0f728ba15b38579efefc3dc47821882ca18ffea7/fs/inode.go#L293
   Implemented in https://review.gerrithub.io/c/hanwen/go-fuse/+/513646/2

[savchenko]

Do you see the same corruption on all synchronised copies or does it look different?

Different. Here is an output from another endpoint, same repository I've mentioned in the first post:

Decrypting master key
fsck: corrupt entry in dir "": "Q42GWQmqmH5GvujP8DOrdg"
fsck: error opening dir "xxxxx/xxxxxx/xxxxx/xxxxx/xxxxx/xxxxxxxx": 2=no such file or directory
fsck: corrupt entry in dir "xxxx/.git": "GGdbtiSuWnU4TaoWMpwPSA.sync-conflict-20210309-161800-K35IS4A"
fsck: corrupt entry in dir "xxxx/.git": "GGdbtiSuWnU4TaoWMpwPSA.sync-conflict-20210309-161819-K35IS4A"
fsck: corrupt entry in dir "xxxx/xxxx/xxxx": "BS1jRx94o2SPZE3VVoKduw"
fsck: corrupt entry in dir "xxxx/xxxx/xxxx/xxxx": "ErCyzALmrA_6iMQ2je18iQ"
fsck: corrupt entry in dir "xxxx/xxxx/xxxx/xxxx/xxxxx": "Q7lbJVgsU7EQJollS_EUng"
fsck: corrupt entry in dir "xxxx/xxxx/xxxxxx": "r3UIaZYYMDQQjIOfM4WpJQ"
fsck: corrupt entry in dir "xxxx/xxxx/xxxxxx": "XHaLuVAw3rTZ0A72ErwQDw"
fsck: corrupt entry in dir "xxxx/xxxx/xxxxxx/xxxxxx": "J9WwFnSWdFMcRtv88muvpg"
fsck: corrupt entry in dir "xxxx/xxxx/xxxxxx/xxxxxx/xxxx": "LKwyeRtRF_0Bd-OWgPEAnA"
fsck: corrupt entry in dir "xxxx/xxxx/xxxxxxx": "nuPcy1D81IzB3sKpQiGSFQ"
fsck: corrupt entry in dir "xxxx/xxxx/xxxxxxx/xxxxxxx": "360a5xjQI5NYzUrCQzyb-A"
fsck: corrupt entry in dir "xxxx/xxxx/xxxxxxx/xxxxxxx": "7Iifijp3w-TGxvt57TVMFw"
fsck: corrupt entry in dir "xxxx/xxxx/xxxxxxx/xxxxxxx": "Wo87J3NGSRa_D7KfTiF-0Q"
fsck: corrupt entry in dir "xxxx/xxxx/xxxxxx": "3cYjofmRbfOomOT_oZvRug"
fsck: corrupt entry in dir "xxxx/xxxx/xxxxxxx/xxxxxx": "fNOALh19-Gk_Nfq3f9q5kQ"
fsck: corrupt entry in dir "xxxx/xxxx/xxxxxxxxx": "ww0XqKavBZqE3xoTXUPJLA"
fsck: corrupt entry in dir "xxxx/xxxx/xxxxxxxxx/xxxxxx": "4NtZBOl8UevnjabTYHT5XA"
fsck: corrupt entry in dir "xxxx/xxxx/xxxxxxxxx": "QsBR3hImqbjav3TYaR7YxQ"
fsck: corrupt entry in dir "xxxx/xxxx/xxxxxxxxx": "Ie_9LDDFJMkoOv2FhKgueg"
fsck summary: 21 corrupt files

[rfjakob]

Thanks! The sync-conflict-20210309-161800-K35IS4A stuff is clearly different, but the rest looks like the same errors with slighly different fsck messages due to different gocryptfs versions?

[tleydxdy]

Similar? problem here. I'm also using syncthing and I realized I can't open the directory with my password anymore. it says cipher: message authentication failed I am certain that the password is correct, since I have a paper backup of it, and the master key on that same paper backup works. the directory is also shared with others and they can't use the password to decrypt it too.

Altho -fsck shows the directory as error free.

I did not have the sharedstorage flag, I didn't know it existed :( I don't think there was concurrent access tho, at least not concurrent writes to the same file

[rfjakob]

@tleydxdy you get this?

Decrypting master key
failed to unlock master key: cipher: message authentication failed
Password incorrect.

Decrypting the master key only depends on gocryptfs.conf being intact, and this file is never modified except when changing the password (gocryptfs -passwd).

Keyboard layout problem maybe?

[tleydxdy]

Keyboard layout problem maybe?
I don't think so, this is pretty strange I agree, but I haven't change devices or anything and this is on mutiple devices between two people. I guess until it happens again it'll only be a fluke (maybe I changed the password while sleeping)