blockstore: OpenReadWrite should use atomic writes

[mvdan]

Right now, OpenReadWrite writes directly to the destination file. This has multiple problems:

• If we encounter some error mid-finalize, we may leave a corrupted file
• If we encounter a write/flush error while finalizing, we may leave a partial file
• If we're resuming on an existing file and writing in-place, we may corrupt the user's data (e.g. Opening car blockstore with different header damages car #247)

It would be much saner overall to instead write to a temporary copy next to the destination file (such as foo.car.tmp for foo.car), and once Finalize has finished with no error, we do a rename that should basically never fail.

This also means that we can teach OpenReadWrite to always remove foo.car.tmp when it's discarded, to ensure we don't leave unfinished business behind.

The method described above should be doable with just a bit of os and io glue. We could use https://pkg.go.dev/github.com/google/renameio#TempFile, though I think pulling in a library for this use case is perhaps overkill. We want to prevent common errors, and we're not that concerned with atomicity between multiple processes - users should not be writing to the same CAR file concurrently.

cc @masih @willscott

[masih]

So would foo.car.tmp be a copy of foo.car while it's being written to and not finalised or discarded?

If so, I wonder what the implications would be when working with large CAR files; this would double the storage demand I think.