Skip to content

Optimize atomic writes in BlobContainer for Cloud providers #30680

Closed
@ywelsch

Description

@ywelsch

Certain repository implementations (e.g. S3) provide all-or-nothing semantics when putting files. The snapshot/restore implementation currently has its own way of achieving atomic writes for certain files, by writing to a temporary filename first, and then moving the file using an atomic rename (supported by some filesystems). The cloud providers don't offer a rename operation, and our Cloud provider implementations achieve this by copying the file, and then deleting the original file. This seems wasteful, as it results in more writes, without providing any additional guarantees. I therefore suggest to introduce an "atomicWriteBlob" method directly on BlobContainer that can be implemented in different ways then by the specific implementations, and use the atomic rename trick for FSBlobStore while just delegating to "writeBlob" for Cloud providers.

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions