Background and motivation

Context; this is part of Epic: IDistributedCache updates in .NET 9 and Hybrid Cache API proposal

Was originally just IDistributedCache, but this issue now updated to include all of the abstract HybridCache API (but not the aspnet implementation)

HybridCache is a new cache abstraction that sits on top of IDistributedCache (L2) and IMemoryCache (L1) to provide an integrated cache experience that includes serialization, stampede protection, and a range of other features. The API has been discussed extensively as part of aspnet, most significantly in the aforementioned dotnet/aspnetcore#54647.

We would now like to kick the aspnet bits over the fence into Microsoft.Extensions.Caching.Abstractions, so that:

• backend implementations such as Redis, SQL, etc do not need an aspnet framework reference
• applications other than aspnet can consume the APIs
• other non-aspnet implementations are possible (in particular, FusionCache have expressed interest)

The specific proposed API changes are laid out in #103103

The existing IDistributedCache API is based around byte[], which is wildly inefficient for anything that isn't an in-memory lookup of string to byte[] (i.e. handing back the same array each time, which is itself a bit dangerous because of array mutation).

To be 100% explicit:

• because the byte[] needs to be right-sized, it must be allocated per usage (especially if we want defensive copies)
• even if we knew the length, to use an array-segment efficiently we would also need to agree a recycling strategy between caller and callee
• byte[] demands contiguous memory, which can force LOH etc

The proposal is to add non-allocating APIs, similar to those used for Output Cache in .NET 8, to avoid these allocations; this assists every other backend - Redis, SQL, SQLite, Cosmos, etc.

As an example of the impact of this, see this table (note also the second table in the same comment), where a mocked up version of the API was used to test a FASTER-based cache backend (this is a useful backend because it has very low internal overheads).

| Method         | KeyLength | PayloadLength | Mean        | Error       | StdDev      | Gen0   | Gen1   | Allocated |
|--------------- |---------- |-------------- |------------:|------------:|------------:|-------:|-------:|----------:|
| Get            | 128       | 10240         |    576.0 ns |     9.79 ns |     5.83 ns | 0.6123 |      - |   10264 B |
| Set            | 128       | 10240         |    882.0 ns |    23.99 ns |    22.44 ns | 0.6123 |      - |   10264 B |
| GetAsync       | 128       | 10240         |    657.6 ns |    16.96 ns |    14.16 ns | 0.6189 |      - |   10360 B |
| SetAsync       | 128       | 10240         |  1,094.7 ns |    55.15 ns |    51.58 ns | 0.6123 |      - |   10264 B |
|                |           |               |             |             |             |        |        |           |
| GetBuffer      | 128       | 10240         |    366.1 ns |     6.22 ns |     5.20 ns |      - |      - |         - |
| SetBuffer      | 128       | 10240         |    495.4 ns |     7.11 ns |     2.54 ns |      - |      - |         - |
| GetAsyncBuffer | 128       | 10240         |    387.9 ns |     7.60 ns |     1.97 ns | 0.0014 |      - |      24 B |
| SetAsyncBuffer | 128       | 10240         |    649.9 ns |    12.70 ns |    11.88 ns |      - |      - |         - |

API Proposal

// add extension API for existing IDistributedCache, to avoid byte[] overheads
namespace Microsoft.Extensions.Caching.Distributed;

public interface IBufferDistributedCache : IDistributedCache
{
    bool TryGet(string key, IBufferWriter<byte> destination);
    ValueTask<bool> TryGetAsync(string key, IBufferWriter<byte> destination, CancellationToken token = default);
    void Set(string key, ReadOnlySequence<byte> value, DistributedCacheEntryOptions options);
    ValueTask SetAsync(string key, ReadOnlySequence<byte> value, DistributedCacheEntryOptions options, CancellationToken token = default);
}

// define abstract API for new HybridCache system
namespace Microsoft.Extensions.Caching.Hybrid;

public abstract class HybridCache
{
    public abstract ValueTask<T> GetOrCreateAsync<TState, T>(string key, TState state, Func<TState, CancellationToken, ValueTask<T>> factory, 
        HybridCacheEntryOptions? options = null, IEnumerable<string>? tags = null, CancellationToken cancellationToken = default);
    public ValueTask<T> GetOrCreateAsync<T>(string key, Func<CancellationToken, ValueTask<T>> factory, 
        HybridCacheEntryOptions? options = null, IEnumerable<string>? tags = null, CancellationToken cancellationToken = default);
    public abstract ValueTask SetAsync<T>(string key, T value, HybridCacheEntryOptions? options = null, IEnumerable<string>? tags = null, CancellationToken cancellationToken = default);
    public abstract ValueTask RemoveAsync(string key, CancellationToken cancellationToken = default);
    public virtual ValueTask RemoveAsync(IEnumerable<string> keys, CancellationToken cancellationToken = default);
    public virtual ValueTask RemoveByTagAsync(IEnumerable<string> tags, CancellationToken cancellationToken = default);
    public abstract ValueTask RemoveByTagAsync(string tag, CancellationToken cancellationToken = default);
}

public sealed class HybridCacheEntryOptions
{
    public TimeSpan? Expiration { get; init; }
    public TimeSpan? LocalCacheExpiration { get; init; }
    public HybridCacheEntryFlags? Flags { get; init; }
}

[Flags]
public enum HybridCacheEntryFlags
{
    None = 0,
    DisableLocalCacheRead = 1 << 0,
    DisableLocalCacheWrite = 1 << 1,
    DisableLocalCache = DisableLocalCacheRead | DisableLocalCacheWrite,
    DisableDistributedCacheRead = 1 << 2,
    DisableDistributedCacheWrite = 1 << 3,
    DisableDistributedCache = DisableDistributedCacheRead | DisableDistributedCacheWrite,
    DisableUnderlyingData = 1 << 4,
    DisableCompression = 1 << 5,
}

public interface IHybridCacheSerializer<T>
{
    T Deserialize(ReadOnlySequence<byte> source);
    void Serialize(T value, IBufferWriter<byte> target);
}

public interface IHybridCacheSerializerFactory
{
    bool TryCreateSerializer<T>([NotNullWhen(true)] out IHybridCacheSerializer<T>? serializer);
}

(edit: changed name from cancellationToken to token to mirror IDistributedCache, and added = default)
(edit: added the sync paths)

API Usage

The usage of this API is optional; existing backends that implement IDistributedCache may choose (or not) to additionally implement the new API. The new "hybrid cache" piece will type-test for the feature, and use it appropriately. Any backends that do not implement the API: continue to work, using the byte[] allocation.

The design for "set" is simple: the caller owns the memory lifetime via ReadOnlySequence<byte> - the backend (as an API contract) is explicitly meant to copy the data out; storing the passed in value is undefined behaviour as that data may go out of scope.

The design for "get" is for the caller to handle memory management (which would otherwise be duplicated and brittle in every backend); this is achieved by passing in an IBufferWriter<byte> to which the backend can push the data. This also means that the "hybrid cache" piece can handle quotas etc before data is fully read. The bool return-value is used to distinguish "found" vs "not found"; this is null vs not null on the old API, and is necessary because zero bytes is a valid payload length in some formats (I'm looking at you, protobuf).

Note that hybrid cache proposal only uses async fetch; however, it is noted that if we only added async methods, this would mean that IBufferDistributedCache is "unbalanced" vs IDistributedCache (sync vs async), and omitting it would limit our options later if we decide to add sync paths on hybrid cache; accordingly, sync get+set paths are included in this proposal.

Alternative Designs

• Stream - allocatey, indirect, and multi-copylicious; significant complications for producer and consumer
• IMemoryOwner<byte> or similar return - contiguous, caller gets no chance to intercept until all prepared
• ReadOnlySpan<byte> input (to avoid storage) - contiguous, only applies to "sync"
• default interface methods rather than new interface - would be a similar API change, but would involve an extra memcpy and lease for each fetch (default implementation would be "get array, write array to buffer-writer, which in turn needs to lease"); it is preferable to type test instead (once at setup), and use the most efficient strategy
• naming... yeah, I'm open to offers

Risks

None; any existing backends not implementing the feature continue to work as current, hopefully adding support in time; there is no additional service registration for this auxiliary API