[API Proposal]: One shot hashing of `Stream`

[vcsjones]

Background and motivation

As of .NET 5 (Hash) and .NET 6 (HMAC) we have static one-shots for computing the hash of bytes, either in the form of an array of a ReadOnlySpan<byte>.

This one shot I think is a good direction for .NET. To re-cap what we have today in terms of APIs:

1. HASHALG.HashData - one shot of fixed-length buffers, array or span
2. IncrementalHash - updatable hash
3. HASHALG.Create - .NET Framework 1.0 design

My personal belief is that number three should be mostly de-emphasized for common use cases. One shots should use option one, and incremental should use two. Create should be there for cases where polymorphic behavior is desired, if ever.

There is one case remaining (I believe) where option one and two don't handle as well as three: hashing streams, because the HashAlgorithm instances have ComputeHash(Stream stream)

It's not uncommon to need to hash a stream all in one go (thus still a one-shot). I would propose that these should also be one shots to further reduce the need for option three.

As a matter of implementation, the goal is not to necessarily use platform APIs to produce the most optimal hashing API (though we can certainly do our best to optimize some certain situations). Rather the goal is to provide an API surface that does not require developers to manage an instance of a hash object.

API Proposal

namespace System.Security.Cryptography {
    public abstract partial class MD5 {
        public static byte[] HashData(Stream source);
        public static int HashData(Stream source, Span<byte> destination);

        public static async ValueTask<byte[]> HashDataAsync(Stream source, CancellationToken cancellationToken = default);
        public static async ValueTask<int> HashDataAsync(Stream source, Memory<byte> destination, CancellationToken cancellationToken = default);
    }

    public abstract partial class SHA1 {
        public static byte[] HashData(Stream source);
        public static int HashData(Stream source, Span<byte> destination);

        public static async ValueTask<byte[]> HashDataAsync(Stream source, CancellationToken cancellationToken = default);
        public static async ValueTask<int> HashDataAsync(Stream source, Memory<byte> destination, CancellationToken cancellationToken = default);
    }

    public abstract partial class SHA256 {
        public static byte[] HashData(Stream source);
        public static int HashData(Stream source, Span<byte> destination);

        public static async ValueTask<byte[]> HashDataAsync(Stream source, CancellationToken cancellationToken = default);
        public static async ValueTask<int> HashDataAsync(Stream source, Memory<byte> destination, CancellationToken cancellationToken = default);
    }

    public abstract partial class SHA384 {
        public static byte[] HashData(Stream source);
        public static int HashData(Stream source, Span<byte> destination);

        public static async ValueTask<byte[]> HashDataAsync(Stream source, CancellationToken cancellationToken = default);
        public static async ValueTask<int> HashDataAsync(Stream source, Memory<byte> destination, CancellationToken cancellationToken = default);
    }

    public abstract partial class SHA512 {
        public static byte[] HashData(Stream source);
        public static int HashData(Stream source, Span<byte> destination);

        public static async ValueTask<byte[]> HashDataAsync(Stream source, CancellationToken cancellationToken = default);
        public static async ValueTask<int> HashDataAsync(Stream source, Memory<byte> destination, CancellationToken cancellationToken = default);
    }

    public abstract partial class HMACMD5 {
        public static byte[] HashData(ReadOnlySpan<byte> key, Stream source);
        public static int HashData(ReadOnlySpan<byte> key, Stream source, Span<byte> destination);

        public static async ValueTask<byte[]> HashDataAsync(ReadOnlyMemory<byte> key, Stream source, CancellationToken cancellationToken = default);
        public static async ValueTask<int> HashDataAsync(ReadOnlyMemory<byte> key, Stream source, Memory<byte> destination, CancellationToken cancellationToken = default);
    }

    public abstract partial class HMACSHA1 {
        public static byte[] HashData(ReadOnlySpan<byte> key, Stream source);
        public static int HashData(ReadOnlySpan<byte> key, Stream source, Span<byte> destination);

        public static async ValueTask<byte[]> HashDataAsync(ReadOnlyMemory<byte> key, Stream source, CancellationToken cancellationToken = default);
        public static async ValueTask<int> HashDataAsync(ReadOnlyMemory<byte> key, Stream source, Memory<byte> destination, CancellationToken cancellationToken = default);
    }

    public abstract partial class HMACSHA256 {
        public static byte[] HashData(ReadOnlySpan<byte> key, Stream source);
        public static int HashData(ReadOnlySpan<byte> key, Stream source, Span<byte> destination);

        public static async ValueTask<byte[]> HashDataAsync(ReadOnlyMemory<byte> key, Stream source, CancellationToken cancellationToken = default);
        public static async ValueTask<int> HashDataAsync(ReadOnlyMemory<byte> key, Stream source, Memory<byte> destination, CancellationToken cancellationToken = default);
    }

    public abstract partial class HMACSHA384 {
        public static byte[] HashData(ReadOnlySpan<byte> key, Stream source);
        public static int HashData(ReadOnlySpan<byte> key, Stream source, Span<byte> destination);

        public static async ValueTask<byte[]> HashDataAsync(ReadOnlyMemory<byte> key, Stream source, CancellationToken cancellationToken = default);
        public static async ValueTask<int> HashDataAsync(ReadOnlyMemory<byte> key, Stream source, Memory<byte> destination, CancellationToken cancellationToken = default);
    }

    public abstract partial class HMACSHA512 {
        public static byte[] HashData(ReadOnlySpan<byte> key, Stream source);
        public static int HashData(ReadOnlySpan<byte> key, Stream source, Span<byte> destination);

        public static async ValueTask<byte[]> HashDataAsync(ReadOnlyMemory<byte> key, Stream source, CancellationToken cancellationToken = default);
        public static async ValueTask<int> HashDataAsync(ReadOnlyMemory<byte> key, Stream source, Memory<byte> destination, CancellationToken cancellationToken = default);
    }
}

API Usage

static void Example1() {
    using (FileStream fs = File.Open("/please/hash/me", FileMode.Open)) {
        byte[] fileHash = SHA256.HashData(fs);
    }
}

static void Example2() {
    using (FileStream fs = File.Open("/please/hash/me", FileMode.Open)) {
        Span<byte> buffer = stackalloc byte[32];
        int written = SHA256.TryHashData(fs, buffer);
    }
}

static async Task Example3() {
    using (FileStream fs = File.Open("/please/hash/me", FileMode.Open)) {
        byte[] fileHash = await SHA256.HashDataAsync(fs);
    }
}

static async Task Example4() {
    using (FileStream fs = File.Open("/please/hash/me", FileMode.Open)) {
        Memory<byte> existingBuffer = default; // Something reasonable here.
        int written = await SHA256.TryHashDataAsync(fs, existingBuffer);
    }
}

Alternative Designs

"Do nothing" is likely the best alternative to this. It's possible to do this today using IncrementalHash (read from the stream, update, done) or HASHALG.ComputeHash{Async}. Both of these options however require managing an instance of the hash object.

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

Tagging subscribers to this area: @dotnet/area-system-security, @vcsjones, @krwq
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Background and motivation

As of .NET 5 (Hash) and .NET 6 (HMAC) we have static one-shots for computing the hash of bytes, either in the form of an array of a ReadOnlySpan<byte>.

This one shot I think is a good direction for .NET. To re-cap what we have today in terms of APIs:

1. HASHALG.HashData - one shot of fixed-length buffers, array or span
2. IncrementalHash - updatable hash
3. HASHALG.Create - .NET Framework 1.0 design

My personal belief is that number three should be mostly de-emphasized for common use cases. One shots should use option one, and incremental should use two. Create should be there for cases where polymorphic behavior is desired, if ever.

There is one case remaining (I believe) where option one and two don't handle as well as three: hashing streams.

It's not uncommon to need to hash a stream all in one go (thus still a one-shot). I would propose that these should also be one shots to further reduce the need for option three.

As a matter of implementation, the goal is not to necessarily use platform APIs to produce the most optimal hashing API (though we can certainly do our best to optimize some certain situations). Rather the goal is to provide an API surface that does not require developers to manage an instance of a hash object.

API Proposal

namespace System.Security.Cryptography {
    public abstract partial class MD5 {
        public static byte[] HashData(Stream source);
        public static int TryHashData(Stream source, Span<byte> destination);

        public static Task<byte[]> HashDataAsync(Stream source, CancellationToken cancellationToken = default);
        public static Task<int> TryHashDataAsync(Stream source, Memory<byte> destination, CancellationToken cancellationToken = default);
    }

    public abstract partial class SHA1 {
        public static byte[] HashData(Stream source);
        public static int HashData(Stream source, Span<byte> destination);

        public static Task<byte[]> HashDataAsync(Stream source, CancellationToken cancellationToken = default);
        public static Task<int> TryHashDataAsync(Stream source, Memory<byte> destination, CancellationToken cancellationToken = default);
    }

    public abstract partial class SHA256 {
        public static byte[] HashData(Stream source);
        public static int TryHashData(Stream source, Span<byte> destination);

        public static Task<byte[]> HashDataAsync(Stream source, CancellationToken cancellationToken = default);
        public static Task<int> TryHashDataAsync(Stream source, Memory<byte> destination, CancellationToken cancellationToken = default);
    }

    public abstract partial class SHA384 {
        public static byte[] HashData(Stream source);
        public static int TryHashData(Stream source, Span<byte> destination);

        public static Task<byte[]> HashDataAsync(Stream source, CancellationToken cancellationToken = default);
        public static Task<int> TryHashDataAsync(Stream source, Memory<byte> destination, CancellationToken cancellationToken = default);
    }

    public abstract partial class SHA512 {
        public static byte[] HashData(Stream source);
        public static int TryHashData(Stream source, Span<byte> destination);

        public static Task<byte[]> HashDataAsync(Stream source, CancellationToken cancellationToken = default);
        public static Task<int> TryHashDataAsync(Stream source, Memory<byte> destination, CancellationToken cancellationToken = default);
    }

    public abstract partial class HMACMD5 {
        public static byte[] HashData(ReadOnlySpan<byte> key, Stream source);
        public static int TryHashData(ReadOnlySpan<byte> key, Stream source, Span<byte> destination);

        public static Task<byte[]> HashDataAsync(ReadOnlyMemory<byte> key, Stream source, CancellationToken cancellationToken = default);
        public static Task<int> TryHashDataAsync(ReadOnlyMemory<byte> key, Stream source, Memory<byte> destination, CancellationToken cancellationToken = default);
    }

    public abstract partial class HMACSHA1 {
        public static byte[] HashData(ReadOnlySpan<byte> key, Stream source);
        public static int TryHashData(ReadOnlySpan<byte> key, Stream source, Span<byte> destination);

        public static Task<byte[]> HashDataAsync(ReadOnlyMemory<byte> key, Stream source, CancellationToken cancellationToken = default);
        public static Task<int> TryHashDataAsync(ReadOnlyMemory<byte> key, Stream source, Memory<byte> destination, CancellationToken cancellationToken = default);
    }

    public abstract partial class HMACSHA256 {
        public static byte[] HashData(ReadOnlySpan<byte> key, Stream source);
        public static int TryHashData(ReadOnlySpan<byte> key, Stream source, Span<byte> destination);

        public static Task<byte[]> HashDataAsync(ReadOnlyMemory<byte> key, Stream source, CancellationToken cancellationToken = default);
        public static Task<int> TryHashDataAsync(ReadOnlyMemory<byte> key, Stream source, Memory<byte> destination, CancellationToken cancellationToken = default);
    }

    public abstract partial class HMACSHA384 {
        public static byte[] HashData(ReadOnlySpan<byte> key, Stream source);
        public static int TryHashData(ReadOnlySpan<byte> key, Stream source, Span<byte> destination);

        public static Task<byte[]> HashDataAsync(ReadOnlyMemory<byte> key, Stream source, CancellationToken cancellationToken = default);
        public static Task<int> TryHashDataAsync(ReadOnlyMemory<byte> key, Stream source, Memory<byte> destination, CancellationToken cancellationToken = default);
    }

    public abstract partial class HMACSHA512 {
        public static byte[] HashData(ReadOnlySpan<byte> key, Stream source);
        public static int TryHashData(ReadOnlySpan<byte> key, Stream source, Span<byte> destination);

        public static Task<byte[]> HashDataAsync(ReadOnlyMemory<byte> key, Stream source, CancellationToken cancellationToken = default);
        public static Task<int> TryHashDataAsync(ReadOnlyMemory<byte> key, Stream source, Memory<byte> destination, CancellationToken cancellationToken = default);
    }
}

API Usage

static void Example1() {
    using (FileStream fs = File.Open("/please/hash/me", FileMode.Open)) {
        byte[] fileHash = SHA256.HashData(fs);
    }
}

static void Example2() {
    using (FileStream fs = File.Open("/please/hash/me", FileMode.Open)) {
        Span<byte> buffer = stackalloc byte[32];
        int written = SHA256.TryHashData(fs, buffer);
    }
}

static async Task Example3() {
    using (FileStream fs = File.Open("/please/hash/me", FileMode.Open)) {
        byte[] fileHash = await SHA256.HashDataAsync(fs);
    }
}

static async Task Example4() {
    using (FileStream fs = File.Open("/please/hash/me", FileMode.Open)) {
        Memory<byte> existingBuffer = default; // Something reasonable here.
        int written = await SHA256.TryHashDataAsync(fs, existingBuffer);
    }
}

Alternative Designs

"Do nothing" is likely the best alternative to this. It's possible to do this today using IncrementalHash (read from the stream, update, done) or HASHALG.ComputeHash{Async}. Both of these options however require managing an instance of the hash object.

Risks

No response

Author:	vcsjones
Assignees:	-
Labels:	api-suggestion, area-System.Security
Milestone:	-

[Tornhoof]

Suggestion: ValueTask, the ReadAsync overload with Memory on Stream is ValueTask too.

[vcsjones]

Suggestion: ValueTask, the ReadAsync overload with Memory on Stream is ValueTask too.

That seems sensible. Updated (I suppose this will be discussed in The API Review, assuming this gets that far).

[GrabYourPitchforks]

+1 to this proposal! Some initial feedback:

• The key should be taken as a ROS<byte> across all overloads for HMAC. There's no need for the implementation to preserve the key material across the async boundary.
• I almost want to get rid of the Try* routines, as there could be ambiguity as to the failure conditions. So that would mean that there are overloads of HashData that take a destination buffer as a parameter. All of these algorithms have a fixed digest size, so we could say that the burden is on the caller to pass an appropriately sized buffer. The method could still return int if desired to make slicing easier, but it's not necessary.

I know the int vs. void thing has been the subject of API review debate. I expect Jeremy has more coherent thoughts on this. IMO this is a minor enough detail that it shouldn't delay review.

[vcsjones]

The key should be taken as a ROS across all overloads for HMAC. There's no need for the implementation to preserve the key material across the async boundary.

Presumably these APIs are going to actually be async. You can't have ref struct parameters on an async method. You will be greeted with:

error CS4012: Parameters or locals of type 'ReadOnlySpan<byte>' cannot be declared in async methods or async lambda expressions.

All of these algorithms have a fixed digest size, so we could say that the burden is on the caller to pass an appropriately sized buffer. The method could still return int if desired to make slicing easier, but it's not necessary.

The existing APIs already return int even though the length of the hash is fixed. I wasn't a huge fan of that, I would have rather have had void. However, these are overloads of other HashData APIs that also return an int. I would favor being consistent.

Looking at my proposal though, the int returning ones should not be Try. They don't return a bool. Try doesn't work well for async since we can't have an out parameter in an async method. So I got rid of the Try cases. If we wanted Trys, it would have to look something like:

public static async Task<(bool Success, int Written)> TryHashDataAsync(
    ReadOnlyMemory<byte> key,
    Stream source) => throw null;

[svick]

@vcsjones

Presumably these APIs are going to actually be async.

I think the method itself does not have to be async. The implementation could look something like this (keep in mind that I don't know anything about cryptography):

ValueTask<byte[]> HashDataAsync(ReadOnlySpan<byte> key, Stream source)
{
    var incrementalHash = IncrementalHash.CreateHMAC(HashAlgorithmName.SHA256, key);

    return HashDataCoreAsync();

    async ValueTask<byte[]> HashDataCoreAsync()
    {
        using (incrementalHash)
        {
            var buffer = new byte[4096];

            while (await source.ReadAsync(buffer) is int bytesRead and not 0)
            {
                incrementalHash.AppendData(buffer.AsSpan(0, bytesRead));
            }

            return incrementalHash.GetHashAndReset();
        }
    }
}

[vcsjones]

@svick

I think the method itself does not have to be async.

Ah. That might work. However, I would say there is one other case where a Memory<byte> might work better, and that is the caller. While we can avoid the async, callers might not be able to. For example:

using System;
using System.IO;
using System.Threading;
using System.Threading.Tasks;

class Program {
    static async Task Main() {
        ReadOnlySpan<byte> key = new byte[32];
        byte[] destination = new byte[64];
        _ = await HMACSHA512.HashDataAsync(key, Stream.Null, destination);
        await File.WriteAllBytesAsync("digest.bin", destination);
    }
}

public abstract partial class HMACSHA512 {
    public static ValueTask<int> HashDataAsync(ReadOnlySpan<byte> key, Stream source, Memory<byte> destination, CancellationToken cancellationToken = default) => await null;
}

That won't compile because Main is async. So, accepting a span seems tricky because async callers won't be able to declare / create spans easily themselves.

[GrabYourPitchforks]

Related (re: span and async): dotnet/csharplang#1331

If in practice we expect callers to have a byte[] for the key, then there's no harm to making the parameter ROS<byte>. The implicit conversion will succeed, same as it would have succeeded for ROM<byte>. Basically, since span is the universal receiver, it should be preferred over memory if practical. Would need to ponder this a bit more.

Of course, I'm waiting until everybody goes on vacation to send the Secret<T> PR, then I can request one-shot overloads that take Secret<byte> key as the first param. 😈

[vcsjones]

in practice we expect callers to have a byte[] for the key

Well in that case I would just say take a byte[] then. The ROM will at least give the opportunity for accepting slices and similar.

then there's no harm to making the parameter ROS. The implicit conversion will succeed

The API shape is.. strange in my opinion. "We accept a ReadOnlySpan<byte> here, but the only way to do it is by implicit conversion". Developers will look at the documentation and left wondering, "how do I even create a span here?"