Repository wrapper for Azure Table Storage in C# using the Azure.Data.Tables libraries and supporting .NET Standard 2.0 and tested in .NET Framework 4.8 and .NET 6.0
Working with Azure Table Storage has been interesting and very different from working with SQL Server which I have done for many years. After reading a number of articles about it and using it I realised a generic wrapper would be useful to aid unit testing and so this is the result of that realisation.
I referenced a number of articles on Table Storage most of which are quite old now but still valid. Suggestions from these articles have been included in this library.
https://azure.microsoft.com/en-gb/blog/managing-concurrency-in-microsoft-azure-storage-2/
https://docs.microsoft.com/en-us/azure/storage/storage-table-design-guide
https://docs.particular.net/nservicebus/azure-storage-persistence/performance-tuning
http://robertgreiner.com/2012/06/why-is-azure-table-storage-so-slow/
Optimisations are controlled by the Table Storage Options Class. The defaults are applied as below if not overridden:
publicclassTableStorageOptions{publicboolUseNagleAlgorithm{get;set;}=false;publicboolExpect100Continue{get;set;}=false;publicintConnectionLimit{get;set;}=10;publicintRetries{get;set;}=3;publicdoubleRetryWaitTimeInSeconds{get;set;}=1;publicboolEnsureTableExists{get;set;}=true;}Example entity:
NOTE: Azure.Data.Tables requires inheritance from the interface as the base class TableEntity is a sealed class.
publicclassTestTableEntity:ITableEntity{publicstringPartitionKey{get;set;}publicstringRowKey{get;set;}publicDateTimeOffset?Timestamp{get;set;}publicETagETag{get;set;}publicintAge{get;set;}publicstringEmail{get;set;}publicTestTableEntity(){}publicTestTableEntity(stringname,stringsurname){PartitionKey=surname;RowKey=name;}}The library also includes a factory class to make it easier when using dependency injection with multiple tables. This can create a table store with the default TableStorageOptions which is used when not specified, or override the options depending on your needs.
publicclassTestTableStorageClient{privateITableStore<MyStuff>_store;publicTestTableStorageClient(ITableStoreFactoryfactory){_store=factory.CreateTableStore<MyStuff>("MyTable","UseDevelopmentStorage=true");}}Support for other credentials other than connection string are also included i.e. TokenCredential, AzureSasCredential and TableSharedKeyCredential
Override TableStorageOptions when using the factory creation
publicclassTestTableStorageClient{privateITableStore<MyStuff>_store;publicTestTableStorageClient(ITableStoreFactoryfactory){varoptions=newTableStorageOptions{UseNagleAlgorithm=true,ConnectionLimit=100,EnsureTableExists=false};_store=factory.CreateTableStore<MyStuff>("MyTable","UseDevelopmentStorage=true",options);}}Override TableStorageOptions when using the store creation
publicclassTestTableStorageClient{privateITableStore<MyStuff>_store;publicTestTableStorageClient(){varoptions=newTableStorageOptions{UseNagleAlgorithm=true,ConnectionLimit=100,EnsureTableExists=false};_store=newTableStore<MyStuff>("MyTable","UseDevelopmentStorage=true",options);}}Example Insert of a record
vartableStorage=newTableStore<TestTableEntity>("MyTable","UseDevelopmentStorage=true");varentity=newTestTableEntity("John","Smith"){Age=21,Email="john.smith@something.com"};awaittableStorage.InsertAsync(entity);// Get the entries by the row keyvarresult=tableStorage.GetByRowKey("John").ToList();Inserting multiple entries into table storage requires each entry to have the same partition key for a batch. This implementation in the wrapper does this job for you so that you can just pass a list of entities.
Example Insert of multiple records
vartableStorage=newTableStore<TestTableEntity>("MyTable","UseDevelopmentStorage=true");varentries=newList<TestTableEntity>{newTestTableEntity("John","Smith"){Age=21,Email="john.smith@something.com"},newTestTableEntity("Jane","Smith"){Age=28,Email="jane.smith@something.com"},newTestTableEntity("Bill","Smith"){Age=38,Email="bill.smith@another.com"},newTestTableEntity("Fred","Jones"){Age=32,Email="fred.jones@somewhere.com"},newTestTableEntity("Bill","Jones"){Age=45,Email="bill.jones@somewhere.com"},newTestTableEntity("Bill","King"){Age=45,Email="bill.king@email.com"},newTestTableEntity("Fred","Bloggs"){Age=32,Email="fred.bloggs@email.com"}};awaittableStorage.InsertAsync(entries);Example of Insert or Replace of a record
vartableStorage=newTableStore<TestTableEntity>("MyTable","UseDevelopmentStorage=true");varentity=newTestTableEntity("John","Smith"){Age=21,Email="john.smith@something.com"};awaittableStorage.InsertOrReplaceAsync(entity);Example of Updating a record
vartableStorage=newTableStore<TestTableEntity>("MyTable","UseDevelopmentStorage=true");// Get the current recordvarentity=awaittableStorage.GetRecordAsync("Smith","John");// Update propertiesentity.Age=22;awaittableStorage.UpdateAsync(entity);Example of deleting a record
vartableStorage=newTableStore<TestTableEntity>("MyTable","UseDevelopmentStorage=true");// Get the current recordvarentity=awaittableStorage.GetRecordAsync("Smith","John");awaittableStorage.DeleteAsync(entity);Example of deleting all records for a partition
vartableStorage=newTableStore<TestTableEntity>("MyTable","UseDevelopmentStorage=true");awaittableStorage.DeleteByPartitionAsync("Smith");Example of deleting all records in all partitions
vartableStorage=newTableStore<TestTableEntity>("MyTable","UseDevelopmentStorage=true");awaittableStorage.DeleteAllAsync();Table Storage does not really have generic way of filtering data as yet. So there are some methods to help with that. NOTE: The filtering works by getting all records so on large datasets this will be slow. Testing showed ~1.3 seconds for 10,000 records Testing when paged by 100 ~0.0300 seconds for 10,000 records returning 100 records
vartableStorage=newTableStore<TestTableEntity>("MyTable","UseDevelopmentStorage=true");varresults=tableStorage.GetRecordsByFilter(x =>x.Age>21&&x.Age<25);And with basic paging starting at 0 and returning 100 NOTE: The start is number of records e.g. 20, 100 would start at record 20 and then return a maxiumum of 100 after that
vartableStorage=newTableStore<TestTableEntity>("MyTable","UseDevelopmentStorage=true");varresults=tableStorage.GetRecordsByFilter(x =>x.Age>21&&x.Age<25,0,100);There is also the consideration of using Reactive Extensions (RX - http://reactivex.io/) to observe the results from a get all records call or a get filtered records.
vartableStorage=newTableStore<TestTableEntity>("MyTable","UseDevelopmentStorage=true");vartheObserver=tableStorage.GetAllRecordsObservable();theObserver.Where(x =>x.Age>21&&x.Age<25).Take(100).Subscribe(x =>{// Do something with the table entry});or
vartableStorage=newTableStore<TestTableEntity>("MyTable","UseDevelopmentStorage=true");vartheObserver=tableStorage.GetRecordsByFilterObservable(x =>x.Age>21&&x.Age<25,0,100);theObserver.Subscribe(x =>{// Do something with the table entry});As Table storage is a schema-less store there are times when you are dealing with multiple entity types in a single table. https://docs.microsoft.com/en-us/azure/cosmos-db/table-storage-design-guide#work-with-heterogeneous-entity-types
This library has some additional support for those times.
When creating a TableStore if no generic type is supplied then it creates a dynamic store. This allows the basic methods, Insert, Update, GetRecord, etc. to specify the generic type on the method call. Getting all records now returns a list of DynamicTableEntity but you can still get by partition key using a generic type.
vartableStorage=newTableStoreDynamic("MyTable","UseDevelopmentStorage=true");varentity=newTestTableEntity("John","Smith"){Age=21,Email="john.smith@something.com"};varotherEntity=newAnotherTableEntity("52a54878-b4b3-45bd-bc5b-3822989b460f","MyProduct"){Name="Product",Url="https://someendpoint"};awaittableStorage.InsertAsync<TestTableEntity>(entity);awaittableStorage.InsertAsync<AnotherTableEntity>(otherEntity);varemployee=awaitGetRecordAsync<TestTableEntity>("Smith","John");varproductEntity=awaitGetRecordAsync<otherEntity>("MyProduct","52a54878-b4b3-45bd-bc5b-3822989b460f");NOTE: Currently only the basic methods are supported for this type of table, there are no filter/search methods.
https://docs.microsoft.com/en-gb/azure/storage/storage-dotnet-how-to-use-tableshttp://www.introtorx.com/content/v1.0.10621.0/01_WhyRx.html
Most methods have a synchronous and asynchronous version.
The unit tests rely on using Azurite Emulator which is now bundled with Visual Studio 2022 details can be found on Microsoft Docs for other installations including Visual Studio Code and Docker.