Vault FS/Git Consistency - Atomic Writes/Consistent Writes in the Lower FS

[joshuakarp]

Created by @CMCDragonkai

The lower fs provides the persistence of writes to our "files" in Polykey. We allow incremental mutation of "files" in Polykey. So that means given a 1 GiB file, editing the middle of the file doesn't mean rewriting the entire file. It's supposed to just edit and mutate that small partition in the middle.

However let's say that the buffer we are writing into the middle is 10 MiB. Then the problem is that when passing 10 MiB into the lower FS which is backed by the Node FS, that results in potentially multiple calls to the write syscall. And multiple write syscalls are not atomic. So that means it is possible to corrupt the encrypted ciphertext backing the plaintext file.

We want to make sure that we have "consistent" updates to our ciphertext files so we don't leave it in an inconsistent state. That would be quite bad.

There are a couple ways of doing this:

1. Write to temporary and perform atomic rename. (This defeats the purpose of incremental mutation)
2. Some sort of COW system

We should look into databases and ACID implementations for some ideas here.

---

It appears that using Git should mean this is not a problem for us. Because we don't care atomicity on a block basis or even on a single ciphertext basis. We only care about atomicity from 1 commit transaction to another. So if something fails in the middle of a commit transaction. An automatic rollback should be made.

---

Resolved in https://gitlab.com/MatrixAI/Engineering/Polykey/js-polykey/-/merge_requests/205

[joshuakarp]

@CMCDragonkai : The ability to rollback inconsistent commit transactions can be introduced in a later version. For now, for the initial release, we can handle potential inconsistent writes as a just a bug.

[CMCDragonkai]

@scottmmorris this is the key point here:

It appears that using Git should mean this is not a problem for us. Because we don't care atomicity on a block basis or even on a single ciphertext basis. We only care about atomicity from 1 commit transaction to another. So if something fails in the middle of a commit transaction. An automatic rollback should be made.

So we need to be able to detect if we have screwed up a write to the Vault. If this occurs, we would want to rollback to an earlier commit since isogit should have kept a snapshot from before.

First thing first is how do we know if there has been a corruption? Does the user get alerted of this? Is there some hashing mechanism we can make use of?

A potential situation is a powerful while changing a secret. At that point the write to the vault may not fully complete.

We need to consider that if we don't have an automatic commit to the isogit, then it's not considered a successful write, and therefore when we launch the vault again, it should reset to the last good commit, no unclean working space is allowed.

[CMCDragonkai]

We should review edgecases for when partial commits occur, when there is dirty data in the working directory of a vault what happens.

[CMCDragonkai]

This is primarily about creating test cases for dirty commits, or partial commits. What happens and how do we ensure that the vault stays clean? I think you encountered a bit of this already @scottmmorris in the vaults refactoring. You can address this issue there too.

[CMCDragonkai]

One idea is to reset the git repository everytime you start the vault. That way at soon as you start js-polykey, the vaults are on a clean state.

[CMCDragonkai]

Should be addressed in https://gitlab.com/MatrixAI/Engineering/Polykey/js-polykey/-/merge_requests/205

[scottmmorris]

It appears iso-git doesn't have a simple way of resetting the repository either through the likes of a git reset or git clean.

This might mean we need to do a recursive walk through the vault directory, checking the status of each file and deleting it if the status is added.

Another possible (but slightly more out there) method would be to clone the repo from itself each time on start up and overwrite it. I'm not even sure this would be possible, but it seems more complex and harder to implement.

I think I will try the first solution now, as we have a recursive walk function already and the expense to resources for this I don't think will be too high.

I'm still trying to work out the best solution for the modification and deletion commit fail scenarios

[CMCDragonkai]

Please read this: https://stackoverflow.com/questions/52704/how-do-i-discard-unstaged-changes-in-git Note that `git checkout`, `git reset` and `git clean` are all porcelain commands. That means they are high level commands that end up using lower level commands. ISOgit may not have those high level analogues, but it doesn't mean that if you don't look at how those commands are actually implemented, you'll be able to do it directly via the isogit library.

…

On 8/3/21 12:41 PM, Scott wrote: It appears iso-git doesn't have a simple way of resetting the repository either through the likes of a |git reset| or |git clean|. This might mean we need to do a recursive walk through the vault directory, checking the status of each file and deleting it if the status is |added|. Another possible (but slightly more out there) method would be to clone the repo from itself each time on start up and overwrite it. I'm not even sure this would be possible, but it seems more complex and harder to implement. I think I will try the first solution now, as we have a recursive walk function already and the expense to resources for this I don't think will be too high. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#180 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAE4OHLMJ3YMRDVT2MTOXPLT25JMVANCNFSM455WWVIA>.

[scottmmorris]

So after testing you I couldn't find a way to use git reset to discard untracked files. The only command that seems to deal with this is git clean. Unfortunately libgit2 also does not specify a clean command along with iso-git.