Skip to content

.NET: chore: support retries on Cosmos storage creation - #402

Merged
Korolev Dmitry (DeagleGross) merged 39 commits into
mainfrom
dmkorolev/cosmos-retries
Aug 21, 2025
Merged

.NET: chore: support retries on Cosmos storage creation#402
Korolev Dmitry (DeagleGross) merged 39 commits into
mainfrom
dmkorolev/cosmos-retries

Conversation

@DeagleGross

@DeagleGrossKorolev Dmitry (DeagleGross) commented Aug 12, 2025

Copy link
Copy Markdown
Contributor

Add configurable retry on Cosmos container creation.

Also includes #425.

Fixes#307
Fixes#305

Contribution Checklist

  • The code builds clean without any errors or warnings
  • The PR follows the Contribution Guidelines
  • All unit tests pass, and I have added new tests where possible
  • I didn't break anyone 😄

@eavanvalkenburgEduard van Valkenburg (eavanvalkenburg) added the .NET Usage: [Issues, PRs], Target: .Net label Aug 12, 2025
@github-actionsgithub-actionsBot changed the title chore: support retries on Cosmos storage creation.NET: chore: support retries on Cosmos storage creationAug 12, 2025
@DeagleGross
Korolev Dmitry (DeagleGross) marked this pull request as ready for review August 15, 2025 11:09
CopilotAI review requested due to automatic review settings August 15, 2025 11:09

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR adds configurable retry functionality to Cosmos DB container creation operations. The implementation addresses transient failures that can occur during container initialization by introducing exponential backoff retry logic with customizable parameters.

Key changes:

  • Introduces a new options class for configuring retry behavior with exponential backoff
  • Modifies the LazyCosmosContainer to support retry logic during container initialization
  • Adds comprehensive test coverage for the new retry functionality

Reviewed Changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

Show a summary per file
FileDescription
CosmosActorStateStorageOptions.csDefines configuration options for retry behavior including max attempts, delays, and backoff multiplier
LazyCosmosContainer.csImplements retry logic with exponential backoff for container initialization operations
ServiceCollectionExtensions.csUpdates dependency injection to pass retry options to LazyCosmosContainer
LazyCosmosContainerTests.csAdds integration test to verify retry configuration is properly applied
Microsoft.Extensions.AI.Agents.Runtime.Storage.CosmosDB.csprojAdds Microsoft.Extensions.Options package reference
Directory.Packages.propsDefines version for Microsoft.Extensions.Options package

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

@adityamandaleeka

Copy link
Copy Markdown
Member

This is an improvement, but it seems like it still can leave the Lazy in a permanently busted state if the bounded number of retries fail.

One way to avoid that is to move to an IAsyncDisposable pattern with an internal retry loop and a CTS. That way the initialization will keep retrying until it succeeds or is canceled.

Example:

internalsealedclassLazyCosmosContainer:IAsyncDisposable{privatereadonlyCancellationTokenSource_cts=new();privateTask<Container>?_initTask;publicTask<Container>GetContainerAsync()=>_initTask??=InitializeWithRetryAsync(_cts.Token);privateasyncTask<Container>InitializeWithRetryAsync(CancellationTokenct){vardelay=TimeSpan.FromSeconds(1);while(true){ct.ThrowIfCancellationRequested();try{returnawaitInitializeContainerAsync();}catch(CosmosExceptionex)when(IsTransient(ex)){awaitTask.Delay(delay,ct);delay=TimeSpan.FromSeconds(Math.Min(delay.TotalSeconds*2,30));}}}publicValueTaskDisposeAsync(){_cts.Cancel();_cts.Dispose();returndefault;}}

@adityamandaleeka

Copy link
Copy Markdown
Member

BTW it might be good to add some jitter in the backoff too so that if multiple instances start at the same time they don't all hammer cosmos in sync.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. We'll need to keep an eye on the CI for a while to make sure it's stable.

Merged via the queue into main with commit 25291deAug 21, 2025
15 checks passed
@DeagleGross
Korolev Dmitry (DeagleGross) deleted the dmkorolev/cosmos-retries branch August 21, 2025 18:40
Reuben Bond (ReubenBond) pushed a commit to ReubenBond/agent-framework that referenced this pull request Oct 28, 2025
* support retries
* tests + registration options
* fix ordering ..
* HK + update packages
* fix paths
* Update dotnet/tests/CosmosDB.IntegrationTests/Microsoft.Extensions.AI.Agents.Runtime.Storage.CosmosDB.Tests/CosmosTestFixture.cs
* re create project and fix some pk usage
* fix all tests
* try workflow?
* wip 1
* fix definition
* try with cosmos_use_emulator env?
* try ignore SSL errors?
* other cert verifications
* hardcode to 8081?
* proper valuation of ENV
* logging
* ensure db exsists for CI
* bump
* cleanup
* fix usage
* nit comment
* try only release for stability?
* try skip some flaky tests
* merge fixes + rollback container
* reimplement with iasyncdisposable pattern
* remove example doc struct
AQ-1970 (Arturo-Quiroga-MSFT) pushed a commit to Arturo-Quiroga-MSFT/agent-framework-public that referenced this pull request Nov 23, 2025
* support retries
* tests + registration options
* fix ordering ..
* HK + update packages
* fix paths
* Update dotnet/tests/CosmosDB.IntegrationTests/Microsoft.Extensions.AI.Agents.Runtime.Storage.CosmosDB.Tests/CosmosTestFixture.cs
* re create project and fix some pk usage
* fix all tests
* try workflow?
* wip 1
* fix definition
* try with cosmos_use_emulator env?
* try ignore SSL errors?
* other cert verifications
* hardcode to 8081?
* proper valuation of ENV
* logging
* ensure db exsists for CI
* bump
* cleanup
* fix usage
* nit comment
* try only release for stability?
* try skip some flaky tests
* merge fixes + rollback container
* reimplement with iasyncdisposable pattern
* remove example doc struct
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

.NETUsage: [Issues, PRs], Target: .Net

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add retries with backoff to Cosmos container initialization .NET: Implement hierarchical partition keys for Cosmos storage impl

5 participants

@DeagleGross@adityamandaleeka@alliscode@eavanvalkenburg