DragonFruit Redis Queue Processor

A lightweight queue processor for C# using redis and json.

Overview

This library aims to provide a lightweight queue processor using a redis sorted set as storage, json to perform serialization and keyspace events to enable distributed processing. It is designed for use in ASP.NET Core projects that use the built-in dependency-injection service. Shutdown signals are respected and processing will aim to finish before shutdown.

If the program shuts down mid-way through processing, any queued tasks will not be lost and will resume once the program has re-initialised.

How to use

1. Install the latest nuget package (see badges above)
2. Enable keyspace events for sorted sets on the redis database

    # redis.conf
   notify-keyspace-events Kz
   # redis cli
   CONFIG SET notify-keyspace-events Kz
   

3. Configure and add an IConnectionMultiplexer as a singleton in Program.cs (or Startup.cs on older style projects)
4. For each job type, update the class to inherit Job. Ideally, the class should also be decorated with [JobTypeId("id")] to allow for future refactoring.
5. Configure the queue processors:
   • services.AddQueueProcessor("queue:key", 5) will create a queue processor that will process tasks on the set queue:key for any type and complete jobs in batches of 5.
   • services.AddQueueProcessor<T>("queue:key") will process one task at a time, but only allow the specified type to be processed.
6. Access and push jobs to the queue:

   publicclassTestController:Controller{// for generic processors, the generic type is IQueueJobprivatereadonlyQueueProcessor<IQueueJob>_queue;publicTestController(QueueProcessor<IQueueJob>queue){_queue=queue;}publicasyncTaskQueueJob(){// TestJob impliments IQueueJob and has a parameterless constructorvarnewJob=newTestJob("params");// this also accepts an array of jobs for bulk submission_queue.Enqueue(newJob);}}