[object_store] Support adding MD5 checksum headers to validate object integrity

[tzembo]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

I'd like verify the integrity of uploaded objects (using some kind of checksum) across all three cloud providers.

Currently, the S3 implementation allows setting an AmazonS3 configuration value that attaches a x-amz-checksum-sha256 header to PUT requests against the store. However:

• This is only available for AWS (and SHA256).
  • AWS supports other checksum algorithms (MD5, SHA1, CRC32, CRC32C)
  • Azure supports MD5 and CRC64
  • GCP supports MD5 and CRC32C
• This requires another pass to calculate the checksum value, which the user of this library may already have computed in another context.

Describe the solution you'd like

I'm proposing that we add a Checksum attribute which specifies a ChecksumAlgorithm enum. The value for this attribute would be the base64-encoded checksum value. For now, MD5 can be the only supported checksum algorithm (which all three cloud providers support via the Content-MD5 header). The value for this algorithm is a base64-encoded 128-bit digest.

pub enum Attribute {
    ...
    /// Provides a checksum used to verify object data integrity
    Checksum(ChecksumAlgorithm),
}

pub enum ChecksumAlgorithm {
    MD5,
}

Describe alternatives you've considered

I considered implementing this for more checksum algorithms, but I'm starting with MD5 because it's the only one supported by all three cloud providers. In the future, we could extend this to support additional checksum algorithms (e.g. CRC32C). However:

• stores that do not support a particular error would need to return an error
• stores would ideally be able to "report" their supported checksum algorithms (for usability reasons)
• (AWS) we'd need to figure out how a SHA-256 checksum provided via attributes interacts with the one provided via config

I considered calling the attribute ContentMD5 but decided to make it a bit more generic to support additional checksums in the future.

Additional context

I can put up a PR for this issue.