Offline-first IoT data pipeline for .NET — guaranteed ordered delivery despite connectivity interruptions.
When an IoT device at a branch location loses connectivity to the cloud, what happens to the telemetry it generates?
In most .NET IoT deployments — it is silently dropped.
MQTTnet's ManagedClient reconnects automatically, but there is no built-in mechanism to:
- Buffer messages locally during outages
- Persist them to disk across process restarts
- Replay them in guaranteed chronological order when connectivity returns
In compliance-sensitive environments — financial institution surveillance, healthcare monitoring, industrial safety — that silent loss has real regulatory and operational consequences.
EdgeSyncBuffer provides an offline-first write guarantee:
Write always succeeds locally
↓
Entries buffered in-memory + persisted to disk
↓
Connectivity restored → all entries delivered to cloud
↓
Guaranteed sequence-number ordering — not FIFO, not timestamp order
Clock drift at edge devices during connectivity outages is common. A device that reconnects after an NTP correction may have timestamps that appear out of sequence. Sequence numbers are assigned at write time and are immune to clock drift — ensuring correct ordering regardless of device clock state.
This is the key insight missing from most IoT buffer implementations.
dotnet add package EdgeSync.BufferRequires .NET 8.0 or later.
usingEdgeSync;// 1. Define your telemetry typepublicrecordCameraEvent(stringCameraId,stringEventType,DateTimeTimestamp,floatConfidence):ITimestamped;// 2. Define your cloud upload functionTask<SyncResult>UploadToCloud(IEnumerable<CameraEvent>batch,CancellationTokenct){// Your Azure IoT Hub, AWS IoT Core, or REST API call hereawaitmyCloudService.SendBatchAsync(batch,ct);returnSyncResult.Ok();}// 3. Create the bufferawaitusingvarbuffer=newEdgeSyncBuffer<CameraEvent>(options:newEdgeSyncOptions{MaxCapacity=100_000,BatchSize=500,PersistPath="/var/iot/camera_events.jsonl",// survives restarts},uploadFn:UploadToCloud,logger:newConsoleEdgeSyncLogger());// 4. Write — always succeeds, even when offlineawaitbuffer.WriteAsync(newCameraEvent("CAM001","MotionDetected",DateTime.UtcNow,0.95f));// 5. When connectivity returns — ordered sync happens automaticallyawaitbuffer.OnConnectivityRestoredAsync();// 6. When connectivity is lost — subsequent writes buffer locallybuffer.OnConnectivityLost();varoptions=newEdgeSyncOptions{// Maximum entries in memory before oldest are evictedMaxCapacity=100_000,// default// Entries per cloud upload batchBatchSize=500,// default// JSON Lines file for disk persistence (null = memory-only)PersistPath=null,// default — set a path to enable// Evictions before dead-letter alert firesDeadLetterThreshold=1_000,// default};// Convenience factory methodsvarmemoryOnly=EdgeSyncOptions.Default;varwithDisk=EdgeSyncOptions.WithPersistence("/var/iot/buffer.jsonl");// Raise an alert when entries are being evicted (buffer pressure)buffer.OnDeadLetterThresholdExceeded+=async stats =>{awaitalertService.SendAsync($"IoT buffer under pressure: {stats.EvictionRate}% eviction rate. "+$"Pending: {stats.CurrentPending}");};varstats=buffer.Stats;Console.WriteLine($"Written: {stats.TotalWritten}");Console.WriteLine($"Synced: {stats.TotalSynced}");Console.WriteLine($"Evicted: {stats.TotalEvicted}");Console.WriteLine($"Pending: {stats.CurrentPending}");Console.WriteLine($"SyncRate: {stats.SyncRate}%");Implement IEdgeSyncLogger to route to your existing logging infrastructure:
// With Microsoft.Extensions.LoggingpublicclassMsExtLogger<T>:IEdgeSyncLogger{privatereadonlyILogger<T>_inner;publicMsExtLogger(ILogger<T>logger)=>_inner=logger;publicvoidLogDebug(stringmsg)=>_inner.LogDebug(msg);publicvoidLogInformation(stringmsg)=>_inner.LogInformation(msg);publicvoidLogWarning(stringmsg)=>_inner.LogWarning(msg);publicvoidLogError(stringmsg)=>_inner.LogError(msg);}- Zero dependencies beyond
System.Text.Json(already in .NET) - Append-only — each write is a single file operation
- Human-readable for debugging
- Simple compaction — rewrite file minus synced entries
For high-throughput scenarios, replace with SQLite via Microsoft.Data.Sqlite.
Channel<T> is excellent for producer/consumer pipelines but requires a dedicated consumer task. ConcurrentQueue<T> allows the buffer to be written from any thread without a background consumer — matching IoT scenarios where writes come from device event callbacks on arbitrary threads.
Cloud SDK retries address transient HTTP failures. EdgeSyncBuffer addresses connectivity outages — periods where no network path exists. These are different failure modes requiring different solutions.
Issues and pull requests are welcome. Please open an issue describing the problem before submitting a PR for significant changes.
# Run tests
dotnet test# Build package locally
dotnet pack src/EdgeSyncBuffer/EdgeSync.Buffer.csproj --configuration Release- Offline-First IoT Data Pipelines in .NET — InfoQ
- Building Resilient IoT Infrastructure for Enterprise Deployments — DZone
MIT — see LICENSE
Ram Budithe — Lead Software Engineer, Global Security Technology
LinkedIn · GitHub