Extend OnRemove event

[MichaConrad]

Status

Implemented events for the following handles

• System.Runtime.Caching
• Microsoft.Extensions.Caching.Memory
• DictionaryCache
• System.Web cache
• Redis!

Goals

In 0.9.3 and previous versions, OnRemove only triggers if items are removed manually. With version 0.9.2 it also works if any instance of the same cache triggers this event and transports the event via the backplane.

But one key feature is missing: Trigger OnRemove if a key expires or gets evicted by the underlying cache.
The in-memory cache handles for System.Runtime.Caching and MS.Extensions.Caching do actually support this, but CacheManager does not make use of it to trigger OnRemove yet.

Implementation concepts

For in-memory caches, this is pretty simple because this is usually a build in functionality and already used by CacheManager.

For distributed caches this gets a little bit more tricky.

• For Redis, CacheManager has to register and listen to keyspace notifications, a feature which must be enabled explicitly on the redis server.
  Therefore feature checks might be needed to gracefully disable the event propagation and maybe log warnings on startup (as early as possible)
• For memcached and couchbase there is actually no way to implement this as memcache doesn't have any feature added since the past x years... meh

[Update 2017-03-05]

Implementation details

I decided to separate the existing OnRemove from the new event as a) the events are triggered differently b) the events are triggered by only one cache handle and not by the manager and c) the passed in arguments are slightly different.

There will be a new OnRemoveByHandle event which gets triggered only by one cache handle if the underlying cache either detects an expired key or decides to evict a key because of memory pressure or other reasons.

For the in-memory caches, the implementation was pretty straight forward. Basically, we can listen on callbacks provided by the cache and pass those through.

Redis Specific

For Redis, things get more complicated:

• There is an explicit configuration option to enable the feature
• Can be configured by code (enableKeyspaceNotifications) or json/xml configuration.
• Per default, this will NOT be enabled.
• To use the feature, every redis server CacheManager connects to, must be configured properly:
  Set notify-keyspace-events to at least Exe. E because we only need keyevents, and x and e for expired and evicted events of keys.

If not configured in Redis, CacheManager will not receive any events.

I'm trying to test if the configuration is present, but the run CONFIG commands via the redis client, useAdmin must be set to true! If that's not set, I cannot read the Redis configuration.
If the check detects missing configuration, warnings will be logged. If the check fails on the client level, a debug message will be logged.

Key and Region value limitations

The key stored in Redis holding the CacheItem is build by using <region>:<key> (concatenating region and key separated by :).
Now, for the event feature to work, I have to retrieve the region and key from the actual key stored in redis (which is just a string).
In case the key itself contains : and there might not even be a region set, there would be no way to determine the correct key/region. Same goes for the region, if the region contains :.
Therefore, if the keyevent notification feature is enabled, I will encode the key and region before storing them in Redis (only if the key or region contains :).

Integration with UpdateMode.UP

If UpdateMode is set to Up and not None, CacheManager will remove the cache item from cache handles above the one which triggered the event. This is actually pretty cool to automatically sync even two level caches with in-memory in-front of Redis for example.

Testing

Testing this is pretty hard, I will probably not be able to 100% validate that all the caches trigger evicts correctly, as that depends on memory pressure.
Expire seems to work just great, even in unit tests, see tests