[API Proposal]: Migrate `HybridCache` from aspnet to runtime

[mgravell]

Background and motivation

Context; this is part of Epic: IDistributedCache updates in .NET 9 and Hybrid Cache API proposal

Was originally just IDistributedCache, but this issue now updated to include all of the abstract HybridCache API (but not the aspnet implementation)

HybridCache is a new cache abstraction that sits on top of IDistributedCache (L2) and IMemoryCache (L1) to provide an integrated cache experience that includes serialization, stampede protection, and a range of other features. The API has been discussed extensively as part of aspnet, most significantly in the aforementioned dotnet/aspnetcore#54647.

We would now like to kick the aspnet bits over the fence into Microsoft.Extensions.Caching.Abstractions, so that:

• backend implementations such as Redis, SQL, etc do not need an aspnet framework reference
• applications other than aspnet can consume the APIs
• other non-aspnet implementations are possible (in particular, FusionCache have expressed interest)

The specific proposed API changes are laid out in #103103

---

The existing IDistributedCache API is based around byte[], which is wildly inefficient for anything that isn't an in-memory lookup of string to byte[] (i.e. handing back the same array each time, which is itself a bit dangerous because of array mutation).

To be 100% explicit:

• because the byte[] needs to be right-sized, it must be allocated per usage (especially if we want defensive copies)
• even if we knew the length, to use an array-segment efficiently we would also need to agree a recycling strategy between caller and callee
• byte[] demands contiguous memory, which can force LOH etc

The proposal is to add non-allocating APIs, similar to those used for Output Cache in .NET 8, to avoid these allocations; this assists every other backend - Redis, SQL, SQLite, Cosmos, etc.

As an example of the impact of this, see this table (note also the second table in the same comment), where a mocked up version of the API was used to test a FASTER-based cache backend (this is a useful backend because it has very low internal overheads).

| Method | KeyLength | PayloadLength | Mean | Error | StdDev | Gen0 | Gen1 | Allocated |
|--------------- |---------- |-------------- |------------:|------------:|------------:|-------:|-------:|----------:|
| Get | 128 | 10240 | 576.0 ns | 9.79 ns | 5.83 ns | 0.6123 | - | 10264 B |
| Set | 128 | 10240 | 882.0 ns | 23.99 ns | 22.44 ns | 0.6123 | - | 10264 B |
| GetAsync | 128 | 10240 | 657.6 ns | 16.96 ns | 14.16 ns | 0.6189 | - | 10360 B |
| SetAsync | 128 | 10240 | 1,094.7 ns | 55.15 ns | 51.58 ns | 0.6123 | - | 10264 B |
| | | | | | | | | |
| GetBuffer | 128 | 10240 | 366.1 ns | 6.22 ns | 5.20 ns | - | - | - |
| SetBuffer | 128 | 10240 | 495.4 ns | 7.11 ns | 2.54 ns | - | - | - |
| GetAsyncBuffer | 128 | 10240 | 387.9 ns | 7.60 ns | 1.97 ns | 0.0014 | - | 24 B |
| SetAsyncBuffer | 128 | 10240 | 649.9 ns | 12.70 ns | 11.88 ns | - | - | - |

API Proposal

// add extension API for existing IDistributedCache, to avoid byte[] overheads
namespace Microsoft.Extensions.Caching.Distributed;

public interface IBufferDistributedCache : IDistributedCache
{
 bool TryGet(string key, IBufferWriter<byte> destination);
 ValueTask<bool> TryGetAsync(string key, IBufferWriter<byte> destination, CancellationToken token = default);
 void Set(string key, ReadOnlySequence<byte> value, DistributedCacheEntryOptions options);
 ValueTask SetAsync(string key, ReadOnlySequence<byte> value, DistributedCacheEntryOptions options, CancellationToken token = default);
}

// define abstract API for new HybridCache system
namespace Microsoft.Extensions.Caching.Hybrid;

public abstract class HybridCache
{
 public abstract ValueTask<T> GetOrCreateAsync<TState, T>(string key, TState state, Func<TState, CancellationToken, ValueTask<T>> factory, 
 HybridCacheEntryOptions? options = null, IEnumerable<string>? tags = null, CancellationToken cancellationToken = default);
 public ValueTask<T> GetOrCreateAsync<T>(string key, Func<CancellationToken, ValueTask<T>> factory, 
 HybridCacheEntryOptions? options = null, IEnumerable<string>? tags = null, CancellationToken cancellationToken = default);
 public abstract ValueTask SetAsync<T>(string key, T value, HybridCacheEntryOptions? options = null, IEnumerable<string>? tags = null, CancellationToken cancellationToken = default);
 public abstract ValueTask RemoveAsync(string key, CancellationToken cancellationToken = default);
 public virtual ValueTask RemoveAsync(IEnumerable<string> keys, CancellationToken cancellationToken = default);
 public virtual ValueTask RemoveByTagAsync(IEnumerable<string> tags, CancellationToken cancellationToken = default);
 public abstract ValueTask RemoveByTagAsync(string tag, CancellationToken cancellationToken = default);
}

public sealed class HybridCacheEntryOptions
{
 public TimeSpan? Expiration { get; init; }
 public TimeSpan? LocalCacheExpiration { get; init; }
 public HybridCacheEntryFlags? Flags { get; init; }
}

[Flags]
public enum HybridCacheEntryFlags
{
 None = 0,
 DisableLocalCacheRead = 1 << 0,
 DisableLocalCacheWrite = 1 << 1,
 DisableLocalCache = DisableLocalCacheRead | DisableLocalCacheWrite,
 DisableDistributedCacheRead = 1 << 2,
 DisableDistributedCacheWrite = 1 << 3,
 DisableDistributedCache = DisableDistributedCacheRead | DisableDistributedCacheWrite,
 DisableUnderlyingData = 1 << 4,
 DisableCompression = 1 << 5,
}

public interface IHybridCacheSerializer<T>
{
 T Deserialize(ReadOnlySequence<byte> source);
 void Serialize(T value, IBufferWriter<byte> target);
}

public interface IHybridCacheSerializerFactory
{
 bool TryCreateSerializer<T>([NotNullWhen(true)] out IHybridCacheSerializer<T>? serializer);
}

(edit: changed name from cancellationToken to token to mirror IDistributedCache, and added = default)
(edit: added the sync paths)

API Usage

The usage of this API is optional; existing backends that implement IDistributedCache may choose (or not) to additionally implement the new API. The new "hybrid cache" piece will type-test for the feature, and use it appropriately. Any backends that do not implement the API: continue to work, using the byte[] allocation.

The design for "set" is simple: the caller owns the memory lifetime via ReadOnlySequence<byte> - the backend (as an API contract) is explicitly meant to copy the data out; storing the passed in value is undefined behaviour as that data may go out of scope.

The design for "get" is for the caller to handle memory management (which would otherwise be duplicated and brittle in every backend); this is achieved by passing in an IBufferWriter<byte> to which the backend can push the data. This also means that the "hybrid cache" piece can handle quotas etc before data is fully read. The bool return-value is used to distinguish "found" vs "not found"; this is null vs not null on the old API, and is necessary because zero bytes is a valid payload length in some formats (I'm looking at you, protobuf).

---

Note that hybrid cache proposal only uses async fetch; however, it is noted that if we only added async methods, this would mean that IBufferDistributedCache is "unbalanced" vs IDistributedCache (sync vs async), and omitting it would limit our options later if we decide to add sync paths on hybrid cache; accordingly, sync get+set paths are included in this proposal.

Alternative Designs

• Stream - allocatey, indirect, and multi-copylicious; significant complications for producer and consumer
• IMemoryOwner<byte> or similar return - contiguous, caller gets no chance to intercept until all prepared
• ReadOnlySpan<byte> input (to avoid storage) - contiguous, only applies to "sync"
• default interface methods rather than new interface - would be a similar API change, but would involve an extra memcpy and lease for each fetch (default implementation would be "get array, write array to buffer-writer, which in turn needs to lease"); it is preferable to type test instead (once at setup), and use the most efficient strategy
• naming... yeah, I'm open to offers

Risks

None; any existing backends not implementing the feature continue to work as current, hopefully adding support in time; there is no additional service registration for this auxiliary API

[neon-sunset]

ReadOnlySequence<byte> for SetAsync may be a bit painful. Offering ReadOnlyMemory<byte> overloads would certainly help.

[mgravell]

@neon-sunset painful how? any ReadOnlyMemory<byte> can be a single-segment ReadOnlySequence<byte> for free, but the reverse is not true; allowing us to use non-contiguous memory is a strongly desirable capability for the usage scenarios, giving us much more flexibility; from a consumption side, there is basically no difference - if the consumer needs it linearized, they can do that themselves at the point they need it; or they can defer and handle the chunks independently

as the person who is going to be implementing both sides of this API from the MSFT perspective: I don't see the concern (and I've actively prototyped it, and to emphasize: this is the pattern that we already proved with the output-cache backend)

[Tornhoof]

Any chance to get rid of the String for your new Interface for the key? ReadOnlySpan still does not work in async methods, right?
ReadOnlySequence is weird.

Many implementations probably convert the string to byte arrays anyway.

[mgravell]

string is, I think, unavoidable. It is the lingua franca for cache keys, and is going to be used everywhere else - including at the primary caller input - they're not going to do a complex lease, write, release; they're going to do $"/food/{id}". I did consider ReadOnlyMemory<char>, but it makes everything unnecessarily obtuse. I guess if we had generator support the ROM-char approach might apply, but... I don't think it is the real cost here, and the L1 etc will want string.

As for backends: I wouldn't presume. A database backend might want string; Redis is fine with string, etc.

ReadOnlySequence<byte> may be "weird", but it is absolutely the type I would expect in modern .NET for library-level (not application-level) "BLOB that might be non-trivial size".

[Tornhoof]

Ah, hmpf, GitHub stole the char from my ReadOnlySequence, ReadOnlySequence<char> instead of string would be weird.

[neon-sunset]

Perhaps a ReadOnlyMemory<char>-based key overloads could be offered still? In particular, ReadOnlyMemory<char> would allow allocation-free lookup by interpolated key through passing a ROM<char> from an interpolated handler and then returning its buffer back to the pool. I assume going through UTF-16 is expected tradeoff given IO-bound nature of many access patterns (but if ReadOnlyMemory<byte> for keys is on the table, that would be awesome!).

[mgravell]

Well, in theory I guess we could use ROM-char, and if the input is string, use AsMemory - and reversing that, there is MemoryMarshal.TryGetString. So yes it is possible, and doesn't necessarily involve an additional unnecessary string. It is a relevant design question. I think ROM-byte for the key is too far.

[neon-sunset]

My elevator pitch for ROM<byte>-based key is that IBufferDistributedCache most likely will be serviced by back-ends that talk UTF-8. As of .NET 8, a lot of performance-oriented APIs do provide UTF-8 overloads. With that said, the escape hatch for this could be ROM<char> as it would allow to avoid unnecessary allocation traffic (through pooling the buffers for encoded keys).

[mgravell]

most likely will be serviced by back-ends that talk UTF-8

That is a concern for the backend; the caller doesn't have any insight or interest in the "how"; if UTF8String had landed: it might be a different conversation

[mgravell]

@Tornhoof ah, your comment makes a lot more sense now; yes, ReadOnlySequence<char> key would be very weird: total agreement; we expect keys to be "reasonably" sized, where "reasonable" means at a minimum "fine in a contiguous chunk"