OAuth2 credential providers leak on teardown (CUSTOM_JWT harness / OAuth gateway) — exhausts 50-provider account quota

[aidandaly24]

Description

agentcore deploy registers a managed OAuth2 credential provider as an imperative pre-deploy step (outside the CloudFormation stack) whenever a CUSTOM_JWT harness — or an OAuth-outbound gateway — is configured with --client-id/--client-secret. Tearing the project down with agentcore remove all + agentcore deploy (or remove of the individual resource) destroys the CloudFormation stack but never deletes the OAuth2 credential provider. The provider is orphaned in the customer's account.

Each AWS account is capped at 50 OAuth2 credential providers (Service Quotas code L-431051DC). Because nothing reaps them, repeated create/teardown cycles accumulate orphaned providers until the quota is exhausted, after which every subsequent CUSTOM_JWT/OAuth deploy fails with:

The number of agent identity Oauth2 credential providers in this account has reached its limit

This is the same root cause that recently broke the E2E suite (harness-custom-jwt.test.ts); that side was patched with test-scoped cleanup, but the underlying product behavior — orphaning OAuth2 providers on teardown — is unfixed for real users.

Steps to Reproduce

1. agentcore create --name MyProj --no-agent
2. agentcore add harness --name MyProj --authorizer-type CUSTOM_JWT --discovery-url <url> --allowed-clients <id> --client-id <id> --client-secret <secret> (registers a managed OAuth credential named MyProj-oauth)
3. agentcore deploy --yes → a MyProj-oauth OAuth2 credential provider is created in AgentCore Identity
4. agentcore remove all && agentcore deploy --yes (teardown) → CloudFormation stack is destroyed
5. Observe: aws bedrock-agentcore-control list-oauth2-credential-providers still shows MyProj-oauth. It was never deleted.
6. Repeat the create/deploy/teardown cycle ~50 times (or share an account across many projects/CI) → every CUSTOM_JWT deploy now fails with the quota-limit error above.

Expected Behavior

Teardown (remove all + teardown deploy, and individual resource remove) should delete the managed OAuth2 credential provider it created during deploy — the same way payment credential providers are already reaped — so no resource is orphaned in the customer's account.

Actual Behavior

The OAuth2 credential provider is leaked permanently. Stack destroy does not touch it because it is created imperatively, outside the stack.

Root Cause

OAuth2 credential providers are created imperatively pre-deploy:

• src/cli/operations/deploy/pre-deploy-identity.ts → setupOAuth2Providers → createOAuth2Provider
• src/cli/operations/identity/oauth2-credential-provider.ts → CreateOauth2CredentialProviderCommand
• Managed credential name = computeManagedOAuthCredentialName(resourceName) → `${name}-oauth` (src/cli/primitives/credential-utils.ts)

But the teardown path only cleans up payment credential providers — there is no OAuth2 equivalent:

// src/cli/commands/deploy/actions.ts (~L536)
if (context.isTeardownDeploy) {
 // Clean up imperative payment credential providers (CFN stack delete handles manager/connector/roles).
 // Harnesses are part of the CloudFormation stack, so stack destroy handles them. <-- misleading
 ...
 await cleanupPaymentCredentialProviders({ region: target.region, payments: existingPayments });
 ...
 const teardown = await performStackTeardown(target.name);
}

The comment "Harnesses are part of the CloudFormation stack, so stack destroy handles them" is true for the harness runtime resource but not for its managed OAuth2 credential provider, which lives outside the stack. A grep confirms nothing in src/ calls DeleteOauth2CredentialProvider — only ApiKey and Payment providers have delete paths.

Proposed Fix

Add OAuth2 credential provider cleanup to the teardown path, mirroring cleanupPaymentCredentialProviders:

1. Add a deleteOAuth2Provider / cleanupOAuth2CredentialProviders operation in src/cli/operations/deploy/pre-deploy-identity.ts (or alongside the existing OAuth2 ops in src/cli/operations/identity/oauth2-credential-provider.ts) using DeleteOauth2CredentialProviderCommand.
2. Call it from the isTeardownDeploy branch in src/cli/commands/deploy/actions.ts, iterating the project's OAuthCredentialProvider credentials (resolving names via computeManagedOAuthCredentialName for managed harness/gateway creds). Make it best-effort like payment cleanup (warn, don't fail teardown).
3. Consider the same on individual-resource remove for a harness/gateway that owns a managed OAuth credential, so a single remove doesn't orphan the provider either.
4. Be careful not to delete a provider that another still-deployed resource in the project references (only reap providers owned by the resource(s) being torn down).

Acceptance Criteria

• After agentcore remove all + teardown deploy of a CUSTOM_JWT harness project, list-oauth2-credential-providers shows the <name>-oauth provider is gone.
• Removing an individual harness/gateway that owns a managed OAuth credential deletes its OAuth2 provider.
• Cleanup is best-effort (a delete failure logs a warning and does not abort teardown).
• Cleanup never deletes a provider still referenced by another deployed resource in the project.
• Test coverage for the teardown cleanup path.

Notes

• Quota L-431051DC ("Resource OAuth2 credential providers", default 50) is adjustable, but a limit increase only delays the leak — the durable fix is reaping on teardown.
• Related (test-side mitigation, separate PR): added cleanupStaleOAuth2CredentialProviders to the E2E globalSetup hook and per-test teardown in harness-custom-jwt.test.ts. That keeps CI green but does not fix the product behavior described here.

[notgitika]

🔎 Automated Bug Verification

Confirmed. The teardown path reaps payment credential providers but has no OAuth2 equivalent, so the managed <name>-oauth provider is orphaned on every teardown — exactly the root cause described. The reporter's end-to-end steps need a live AWS account (so I couldn't run steps 1–6 directly), but I reproduced the underlying defect deterministically without AWS and confirmed the mechanism by source inspection.

Root cause (matches the report): OAuth2 providers are created imperatively pre-deploy, outside CloudFormation — createOAuth2Provider → CreateOauth2CredentialProviderCommand (src/cli/operations/identity/oauth2-credential-provider.ts:95-100), driven by setupOAuth2Providers (src/cli/operations/deploy/pre-deploy-identity.ts:358). The teardown branch (src/cli/commands/deploy/actions.ts:557-568, and the TUI src/cli/tui/screens/deploy/useDeployFlow.ts:870) calls only cleanupPaymentCredentialProviders; there is zero OAuth2 delete/cleanup path anywhere in src/. Because the provider lives outside the stack, stack destroy can't reap it — so it survives every remove all + teardown-deploy cycle, accumulating against the 50-provider quota (L-431051DC).

Environment: aws/agentcore-cli @ 257d154 · Node 20 · vitest (no live AWS)

Validated derived reproduction (vitest, 5/5 pass = leak present) + output

import * as identity from '../pre-deploy-identity.js';
import * as oauthOps from '../../identity/oauth2-credential-provider.js';
import * as payments from '../../../aws/agentcore-payments.js';
import { readFileSync } from 'fs';
import { dirname, resolve } from 'path';
import { fileURLToPath } from 'url';
import { describe, expect, it } from 'vitest';

const here = dirname(fileURLToPath(import.meta.url));
const srcRoot = resolve(here, '../../..'); // -> src/cli
const read = (rel: string) => readFileSync(resolve(srcRoot, rel), 'utf8');

describe('OAuth2 credential provider leak on teardown (derived repro)', () => {
 it('PAYMENTS have both a teardown-cleanup fn and a delete op (the working reference)', () => {
 expect(typeof (identity as Record<string, unknown>).cleanupPaymentCredentialProviders).toBe('function');
 expect(typeof (payments as Record<string, unknown>).deletePaymentCredentialProvider).toBe('function');
 });

 it('OAuth2 has NO teardown-cleanup fn (leak: nothing reaps it)', () => {
 const oauthCleanup = Object.keys(identity).filter(
 n => /oauth/i.test(n) && /(cleanup|delete|teardown|remove|reap)/i.test(n)
 );
 expect(oauthCleanup).toEqual([]);
 });

 it('OAuth2 ops module exports create/get/update but NO delete', () => {
 const names = Object.keys(oauthOps);
 expect(names).toContain('createOAuth2Provider');
 expect(names).toContain('getOAuth2Provider');
 expect(names).toContain('updateOAuth2Provider');
 expect(names.filter(n => /(delete|cleanup|remove)/i.test(n))).toEqual([]);
 });

 it('oauth2-credential-provider.ts imports Create/Get/Update but NOT any Delete command', () => {
 const src = read('operations/identity/oauth2-credential-provider.ts');
 expect(src).toContain('CreateOauth2CredentialProviderCommand');
 expect(src).toContain('GetOauth2CredentialProviderCommand');
 expect(src).toContain('UpdateOauth2CredentialProviderCommand');
 expect(src).not.toMatch(/DeleteOauth2CredentialProviderCommand/i);
 });

 it('the deploy teardown branch reaps payments only — OAuth2 is never cleaned up', () => {
 const actions = read('commands/deploy/actions.ts');
 const idx = actions.lastIndexOf('if (context.isTeardownDeploy) {');
 expect(idx).toBeGreaterThan(-1);
 const teardownBlock = actions.slice(idx, idx + 2000);
 expect(teardownBlock).toContain('cleanupPaymentCredentialProviders');
 expect(teardownBlock).not.toMatch(/oauth/i);
 });
});

Output:

 ✓ PAYMENTS have both a teardown-cleanup fn and a delete op (the working reference)
 ✓ OAuth2 has NO teardown-cleanup fn (leak: nothing reaps it)
 ✓ OAuth2 ops module exports create/get/update but NO delete
 ✓ oauth2-credential-provider.ts imports Create/Get/Update but NOT any Delete command
 ✓ the deploy teardown branch reaps payments only — OAuth2 is never cleaned up

 Test Files 1 passed (1)
 Tests 5 passed (5)

A static grep proof corroborates it: DeleteOauth2CredentialProvider (and any oauth+delete/cleanup helper) appears nowhere in src/, while the payment delete path (deletePaymentCredentialProvider, aws/agentcore-payments.ts:330) is present and wired into the teardown branch.

Each assertion flips red the moment a real fix lands (a cleanupOAuth2CredentialProviders reaper + a DeleteOauth2CredentialProviderCommand delete op, invoked from the teardown/remove paths for providers owned by the resource being torn down — per the proposed fix). One scope note: this proves the structural defect (nothing deletes the imperatively-created provider), not the downstream ~50-cycle quota-exhaustion end-state, which needs a live account.

_{Automated triage. This reproduction is a starting point, not a final determination.}