Uh oh!
There was an error while loading. Please reload this page.
.NET: chore: support retries on Cosmos storage creation - #402
Conversation
There was a problem hiding this comment.
Pull Request Overview
This PR adds configurable retry functionality to Cosmos DB container creation operations. The implementation addresses transient failures that can occur during container initialization by introducing exponential backoff retry logic with customizable parameters.
Key changes:
- Introduces a new options class for configuring retry behavior with exponential backoff
- Modifies the LazyCosmosContainer to support retry logic during container initialization
- Adds comprehensive test coverage for the new retry functionality
Reviewed Changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| CosmosActorStateStorageOptions.cs | Defines configuration options for retry behavior including max attempts, delays, and backoff multiplier |
| LazyCosmosContainer.cs | Implements retry logic with exponential backoff for container initialization operations |
| ServiceCollectionExtensions.cs | Updates dependency injection to pass retry options to LazyCosmosContainer |
| LazyCosmosContainerTests.cs | Adds integration test to verify retry configuration is properly applied |
| Microsoft.Extensions.AI.Agents.Runtime.Storage.CosmosDB.csproj | Adds Microsoft.Extensions.Options package reference |
| Directory.Packages.props | Defines version for Microsoft.Extensions.Options package |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
…ft/agent-framework into dmkorolev/cosmos-retries
….Agents.Runtime.Storage.CosmosDB.Tests/CosmosTestFixture.cs
This is an improvement, but it seems like it still can leave the Lazy in a permanently busted state if the bounded number of retries fail. One way to avoid that is to move to an IAsyncDisposable pattern with an internal retry loop and a CTS. That way the initialization will keep retrying until it succeeds or is canceled. Example: internalsealedclassLazyCosmosContainer:IAsyncDisposable{privatereadonlyCancellationTokenSource_cts=new();privateTask<Container>?_initTask;publicTask<Container>GetContainerAsync()=>_initTask??=InitializeWithRetryAsync(_cts.Token);privateasyncTask<Container>InitializeWithRetryAsync(CancellationTokenct){vardelay=TimeSpan.FromSeconds(1);while(true){ct.ThrowIfCancellationRequested();try{returnawaitInitializeContainerAsync();}catch(CosmosExceptionex)when(IsTransient(ex)){awaitTask.Delay(delay,ct);delay=TimeSpan.FromSeconds(Math.Min(delay.TotalSeconds*2,30));}}}publicValueTaskDisposeAsync(){_cts.Cancel();_cts.Dispose();returndefault;}} |
BTW it might be good to add some jitter in the backoff too so that if multiple instances start at the same time they don't all hammer cosmos in sync. |
Aditya Mandaleeka (adityamandaleeka)
left a comment
There was a problem hiding this comment.
Looks good. We'll need to keep an eye on the CI for a while to make sure it's stable.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
* support retries * tests + registration options * fix ordering .. * HK + update packages * fix paths * Update dotnet/tests/CosmosDB.IntegrationTests/Microsoft.Extensions.AI.Agents.Runtime.Storage.CosmosDB.Tests/CosmosTestFixture.cs * re create project and fix some pk usage * fix all tests * try workflow? * wip 1 * fix definition * try with cosmos_use_emulator env? * try ignore SSL errors? * other cert verifications * hardcode to 8081? * proper valuation of ENV * logging * ensure db exsists for CI * bump * cleanup * fix usage * nit comment * try only release for stability? * try skip some flaky tests * merge fixes + rollback container * reimplement with iasyncdisposable pattern * remove example doc struct
* support retries * tests + registration options * fix ordering .. * HK + update packages * fix paths * Update dotnet/tests/CosmosDB.IntegrationTests/Microsoft.Extensions.AI.Agents.Runtime.Storage.CosmosDB.Tests/CosmosTestFixture.cs * re create project and fix some pk usage * fix all tests * try workflow? * wip 1 * fix definition * try with cosmos_use_emulator env? * try ignore SSL errors? * other cert verifications * hardcode to 8081? * proper valuation of ENV * logging * ensure db exsists for CI * bump * cleanup * fix usage * nit comment * try only release for stability? * try skip some flaky tests * merge fixes + rollback container * reimplement with iasyncdisposable pattern * remove example doc struct
Add configurable retry on Cosmos container creation.
Also includes #425.
Fixes#307
Fixes#305
Contribution Checklist