fix: correct AB test execution role IAM policy and promote stability

[jariy17]

Summary

• IAM policy — Aligns the auto-created AB test execution role with the official docs, fixing a 400 "Access denied when validating evaluator" error on every deploy with a custom evaluator
• Promote stability — promote ab-test waits for executionStatus === 'RUNNING' before stopping, avoiding 409 errors when promote runs immediately after resume; throws a clear error if the test never reaches RUNNING
• IAM — DescribeLogGroups — Split into its own statement with Resource: "*" (AWS requirement)
• E2E coverage — Added AB test reaches RUNNING status after deploy to ab-test-target-based.test.ts to catch IAM permission regressions post-deploy
• Status test — Fixed incorrect http-gateway resourceType assertion; replaced with ab-test + invocationUrl check
• Local e2e script — Added scripts/run-e2e-local.sh to replicate the GitHub Actions e2e workflow locally

IAM changes

Trust policy — added SourceAccount + SourceArn conditions to prevent confused deputy attacks:

"Condition": {
"StringEquals": { "aws:SourceAccount": "<accountId>" },
"ArnLike": { "aws:SourceArn": "arn:aws:bedrock-agentcore:*:<accountId>:ab-test/*" }
}

Permissions — 3-statement policy aligned with docs:

• AgentCoreResources — all bedrock-agentcore actions scoped to account with ResourceAccount condition (adds GetEvaluator, GetGatewayTarget, ListGatewayTargets, ListConfigurationBundleVersions)
• CloudWatchLogsDescribe — logs:DescribeLogGroups with Resource: "*" (required by AWS)
• CloudWatchLogs — log read/write actions scoped to evaluation and spans log groups

Running E2E tests locally

export E2E_ROLE_ARN=arn:aws:iam::<account>:role/<role>export E2E_SECRET_ARN=arn:aws:secretsmanager:<region>:<account>:secret:<name>
./scripts/run-e2e-local.sh e2e-tests/ab-test-target-based.test.ts

Test plan

• e2e-tests/ab-test-target-based.test.ts — all steps pass including RUNNING check
• e2e-tests/ab-test-config-bundle.test.ts — passes (RUNNING check deferred to follow-up once real config bundles are used)
• Unit tests src/cli/operations/deploy/__tests__/post-deploy-ab-tests.test.ts — 23 passing
• Unit tests src/cli/commands/pause/__tests__/promote.test.ts — 4 passing

[agentcore-cli-automation]

The new EvaluatorReadStatement is scoped to Object.values(deployedResources?.evaluators ?? {}), which only contains custom evaluators deployed by this project. This will leave the original 400 error in place whenever the AB test's online-eval config references evaluators that aren't in deployedResources.evaluators:

• Builtin evaluators (Builtin.Faithfulness, Builtin.GoalSuccessRate, etc.) — these are first-class evaluator references in OnlineEvalConfig.evaluators (see src/schema/schemas/primitives/online-eval-config.ts:23-24 and the handling in AgentCoreOnlineEvaluationConfig.ts:143-161). If the service validates evaluator ARNs uniformly using the execution role, Builtins will fail the same check the PR is trying to fix.
• External evaluator ARNs — OnlineEvalConfig.evaluators also accepts raw ARNs pointing at evaluators outside this project / another account. Those are never in deployedResources.evaluators either.

Separately, even for the custom-evaluator case, the policy grants GetEvaluator on every evaluator in the project rather than just the ones this specific AB test transitively references via its onlineEvalConfigArns. That's a least-privilege concern but not a functional blocker.

A couple of ways to address this:

1. Resolve evaluators transitively from the referenced online eval configs. For each online eval config referenced by testSpec.evaluationConfig, look up its evaluators list in projectSpec.onlineEvalConfigs, and for each entry:

   • Builtin.* → construct the builtin ARN (arn:${partition}:bedrock-agentcore:::evaluator/Builtin.*)
   • arn:... → use as-is
   • otherwise → look up in deployedResources.evaluators

   Pass that de-duplicated list as evaluatorArns. This fixes both the Builtin/external case and the over-broad scoping.

2. If tightening the scope is risky, at minimum add arn:${partition}:bedrock-agentcore:::evaluator/Builtin.* to the Resource list (and optionally arn:${partition}:bedrock-agentcore:${region}:${accountId}:evaluator/*) so users with builtin evaluators don't hit the same 400.

Could you also confirm whether you tested the fix with a Builtin evaluator in the online eval config, or only with a custom one? The test plan only mentions ab-test-target-based.test.ts / ab-test-config-bundle.test.ts — knowing which evaluator types those exercise would help.

[github-actions]

Coverage Report

Status	Category	Percentage	Covered / Total
🔵	Lines	43.11%	9016 / 20912
🔵	Statements	42.4%	9575 / 22582
🔵	Functions	39.96%	1555 / 3891
🔵	Branches	39.98%	5808 / 14527

Generated in workflow #2425 for commit 5b72ab3 by the Vitest Coverage Report Action

[jariy17]

Addressed in 44bc3f2. The evaluator ARN resolution is now transitive from the online eval configs the AB test references — handling all three cases:

• Custom evaluators — looked up from deployedResources.evaluators
• Builtin.* evaluators — ARN constructed as arn:${partition}:bedrock-agentcore:::evaluator/Builtin.*
• External ARN references — used as-is

The policy is now scoped to only the evaluators this specific AB test transitively references (not all project evaluators), addressing the least-privilege concern.

The e2e tests exercise a custom evaluator (ABTestEvaluator in ab-test-target-based.test.ts). We haven't explicitly tested with Builtin evaluators, but the ARN construction follows the same pattern used in the L3 CDK constructs (AgentCoreOnlineEvaluationConfig.ts:143-161).

[notgitika]

waitForRunningThenStop fixes the CLI promote race condition, but if there is a TUI path that calls updateABTest({ executionStatus: "STOPPED" }) directly, it has the same bug right?

[jariy17]

Summary

• IAM policy — Aligns the auto-created AB test execution role with the official docs, fixing a 400 "Access denied when validating evaluator" error on every deploy with a custom evaluator
• Promote stability — promote ab-test waits for executionStatus === 'RUNNING' before stopping, avoiding 409 errors when promote runs immediately after resume; throws a clear error if the test never reaches RUNNING
• IAM — DescribeLogGroups — Split into its own statement with Resource: "*" (AWS requirement)
• E2E coverage — Added AB test reaches RUNNING status after deploy to ab-test-target-based.test.ts to catch IAM permission regressions post-deploy
• Status test — Fixed incorrect http-gateway resourceType assertion; replaced with ab-test + invocationUrl check
• Local e2e script — Added scripts/run-e2e-local.sh to replicate the GitHub Actions e2e workflow locally

IAM changes

Trust policy — added SourceAccount + SourceArn conditions to prevent confused deputy attacks:

"Condition": {
"StringEquals": { "aws:SourceAccount": "<accountId>" },
"ArnLike": { "aws:SourceArn": "arn:aws:bedrock-agentcore:*:<accountId>:ab-test/*" }
}

Permissions — 3-statement policy aligned with docs:

• AgentCoreResources — all bedrock-agentcore actions scoped to account with ResourceAccount condition (adds GetEvaluator, GetGatewayTarget, ListGatewayTargets, ListConfigurationBundleVersions)
• CloudWatchLogsDescribe — logs:DescribeLogGroups with Resource: "*" (required by AWS)
• CloudWatchLogs — log read/write actions scoped to evaluation and spans log groups

Running E2E tests locally

export E2E_ROLE_ARN=arn:aws:iam::<account>:role/<role>export E2E_SECRET_ARN=arn:aws:secretsmanager:<region>:<account>:secret:<name>
./scripts/run-e2e-local.sh e2e-tests/ab-test-target-based.test.ts

Test plan

• e2e-tests/ab-test-target-based.test.ts — all steps pass including RUNNING check
• e2e-tests/ab-test-config-bundle.test.ts — passes (RUNNING check deferred to follow-up once real config bundles are used)
• Unit tests src/cli/operations/deploy/__tests__/post-deploy-ab-tests.test.ts — 23 passing
• Unit tests src/cli/commands/pause/__tests__/promote.test.ts — 4 passing

[agentcore-cli-automation]

The new EvaluatorReadStatement is scoped to Object.values(deployedResources?.evaluators ?? {}), which only contains custom evaluators deployed by this project. This will leave the original 400 error in place whenever the AB test's online-eval config references evaluators that aren't in deployedResources.evaluators:

• Builtin evaluators (Builtin.Faithfulness, Builtin.GoalSuccessRate, etc.) — these are first-class evaluator references in OnlineEvalConfig.evaluators (see src/schema/schemas/primitives/online-eval-config.ts:23-24 and the handling in AgentCoreOnlineEvaluationConfig.ts:143-161). If the service validates evaluator ARNs uniformly using the execution role, Builtins will fail the same check the PR is trying to fix.
• External evaluator ARNs — OnlineEvalConfig.evaluators also accepts raw ARNs pointing at evaluators outside this project / another account. Those are never in deployedResources.evaluators either.

Separately, even for the custom-evaluator case, the policy grants GetEvaluator on every evaluator in the project rather than just the ones this specific AB test transitively references via its onlineEvalConfigArns. That's a least-privilege concern but not a functional blocker.

A couple of ways to address this:

1. Resolve evaluators transitively from the referenced online eval configs. For each online eval config referenced by testSpec.evaluationConfig, look up its evaluators list in projectSpec.onlineEvalConfigs, and for each entry:

   • Builtin.* → construct the builtin ARN (arn:${partition}:bedrock-agentcore:::evaluator/Builtin.*)
   • arn:... → use as-is
   • otherwise → look up in deployedResources.evaluators

   Pass that de-duplicated list as evaluatorArns. This fixes both the Builtin/external case and the over-broad scoping.

2. If tightening the scope is risky, at minimum add arn:${partition}:bedrock-agentcore:::evaluator/Builtin.* to the Resource list (and optionally arn:${partition}:bedrock-agentcore:${region}:${accountId}:evaluator/*) so users with builtin evaluators don't hit the same 400.

Could you also confirm whether you tested the fix with a Builtin evaluator in the online eval config, or only with a custom one? The test plan only mentions ab-test-target-based.test.ts / ab-test-config-bundle.test.ts — knowing which evaluator types those exercise would help.

[github-actions]

Coverage Report

Status	Category	Percentage	Covered / Total
🔵	Lines	43.11%	9016 / 20912
🔵	Statements	42.4%	9575 / 22582
🔵	Functions	39.96%	1555 / 3891
🔵	Branches	39.98%	5808 / 14527

Generated in workflow #2425 for commit 5b72ab3 by the Vitest Coverage Report Action

[jariy17]

Addressed in 44bc3f2. The evaluator ARN resolution is now transitive from the online eval configs the AB test references — handling all three cases:

• Custom evaluators — looked up from deployedResources.evaluators
• Builtin.* evaluators — ARN constructed as arn:${partition}:bedrock-agentcore:::evaluator/Builtin.*
• External ARN references — used as-is

The policy is now scoped to only the evaluators this specific AB test transitively references (not all project evaluators), addressing the least-privilege concern.

The e2e tests exercise a custom evaluator (ABTestEvaluator in ab-test-target-based.test.ts). We haven't explicitly tested with Builtin evaluators, but the ARN construction follows the same pattern used in the L3 CDK constructs (AgentCoreOnlineEvaluationConfig.ts:143-161).

[notgitika]

waitForRunningThenStop fixes the CLI promote race condition, but if there is a TUI path that calls updateABTest({ executionStatus: "STOPPED" }) directly, it has the same bug right?

[jariy17]

Summary

• IAM policy — Aligns the auto-created AB test execution role with the official docs, fixing a 400 "Access denied when validating evaluator" error on every deploy with a custom evaluator
• Promote stability — promote ab-test waits for executionStatus === 'RUNNING' before stopping, avoiding 409 errors when promote runs immediately after resume; throws a clear error if the test never reaches RUNNING
• IAM — DescribeLogGroups — Split into its own statement with Resource: "*" (AWS requirement)
• E2E coverage — Added AB test reaches RUNNING status after deploy to ab-test-target-based.test.ts to catch IAM permission regressions post-deploy
• Status test — Fixed incorrect http-gateway resourceType assertion; replaced with ab-test + invocationUrl check
• Local e2e script — Added scripts/run-e2e-local.sh to replicate the GitHub Actions e2e workflow locally

IAM changes

Trust policy — added SourceAccount + SourceArn conditions to prevent confused deputy attacks:

"Condition": {
"StringEquals": { "aws:SourceAccount": "<accountId>" },
"ArnLike": { "aws:SourceArn": "arn:aws:bedrock-agentcore:*:<accountId>:ab-test/*" }
}

Permissions — 3-statement policy aligned with docs:

• AgentCoreResources — all bedrock-agentcore actions scoped to account with ResourceAccount condition (adds GetEvaluator, GetGatewayTarget, ListGatewayTargets, ListConfigurationBundleVersions)
• CloudWatchLogsDescribe — logs:DescribeLogGroups with Resource: "*" (required by AWS)
• CloudWatchLogs — log read/write actions scoped to evaluation and spans log groups

Running E2E tests locally

export E2E_ROLE_ARN=arn:aws:iam::<account>:role/<role>export E2E_SECRET_ARN=arn:aws:secretsmanager:<region>:<account>:secret:<name>
./scripts/run-e2e-local.sh e2e-tests/ab-test-target-based.test.ts

Test plan

• e2e-tests/ab-test-target-based.test.ts — all steps pass including RUNNING check
• e2e-tests/ab-test-config-bundle.test.ts — passes (RUNNING check deferred to follow-up once real config bundles are used)
• Unit tests src/cli/operations/deploy/__tests__/post-deploy-ab-tests.test.ts — 23 passing
• Unit tests src/cli/commands/pause/__tests__/promote.test.ts — 4 passing

[agentcore-cli-automation]

The new EvaluatorReadStatement is scoped to Object.values(deployedResources?.evaluators ?? {}), which only contains custom evaluators deployed by this project. This will leave the original 400 error in place whenever the AB test's online-eval config references evaluators that aren't in deployedResources.evaluators:

• Builtin evaluators (Builtin.Faithfulness, Builtin.GoalSuccessRate, etc.) — these are first-class evaluator references in OnlineEvalConfig.evaluators (see src/schema/schemas/primitives/online-eval-config.ts:23-24 and the handling in AgentCoreOnlineEvaluationConfig.ts:143-161). If the service validates evaluator ARNs uniformly using the execution role, Builtins will fail the same check the PR is trying to fix.
• External evaluator ARNs — OnlineEvalConfig.evaluators also accepts raw ARNs pointing at evaluators outside this project / another account. Those are never in deployedResources.evaluators either.

Separately, even for the custom-evaluator case, the policy grants GetEvaluator on every evaluator in the project rather than just the ones this specific AB test transitively references via its onlineEvalConfigArns. That's a least-privilege concern but not a functional blocker.

A couple of ways to address this:

1. Resolve evaluators transitively from the referenced online eval configs. For each online eval config referenced by testSpec.evaluationConfig, look up its evaluators list in projectSpec.onlineEvalConfigs, and for each entry:

   • Builtin.* → construct the builtin ARN (arn:${partition}:bedrock-agentcore:::evaluator/Builtin.*)
   • arn:... → use as-is
   • otherwise → look up in deployedResources.evaluators

   Pass that de-duplicated list as evaluatorArns. This fixes both the Builtin/external case and the over-broad scoping.

2. If tightening the scope is risky, at minimum add arn:${partition}:bedrock-agentcore:::evaluator/Builtin.* to the Resource list (and optionally arn:${partition}:bedrock-agentcore:${region}:${accountId}:evaluator/*) so users with builtin evaluators don't hit the same 400.

Could you also confirm whether you tested the fix with a Builtin evaluator in the online eval config, or only with a custom one? The test plan only mentions ab-test-target-based.test.ts / ab-test-config-bundle.test.ts — knowing which evaluator types those exercise would help.

[github-actions]

Coverage Report

Status	Category	Percentage	Covered / Total
🔵	Lines	43.11%	9016 / 20912
🔵	Statements	42.4%	9575 / 22582
🔵	Functions	39.96%	1555 / 3891
🔵	Branches	39.98%	5808 / 14527

Generated in workflow #2425 for commit 5b72ab3 by the Vitest Coverage Report Action

[jariy17]

Addressed in 44bc3f2. The evaluator ARN resolution is now transitive from the online eval configs the AB test references — handling all three cases:

• Custom evaluators — looked up from deployedResources.evaluators
• Builtin.* evaluators — ARN constructed as arn:${partition}:bedrock-agentcore:::evaluator/Builtin.*
• External ARN references — used as-is

The policy is now scoped to only the evaluators this specific AB test transitively references (not all project evaluators), addressing the least-privilege concern.

The e2e tests exercise a custom evaluator (ABTestEvaluator in ab-test-target-based.test.ts). We haven't explicitly tested with Builtin evaluators, but the ARN construction follows the same pattern used in the L3 CDK constructs (AgentCoreOnlineEvaluationConfig.ts:143-161).

[notgitika]

waitForRunningThenStop fixes the CLI promote race condition, but if there is a TUI path that calls updateABTest({ executionStatus: "STOPPED" }) directly, it has the same bug right?

[jariy17]

Summary

• IAM policy — Aligns the auto-created AB test execution role with the official docs, fixing a 400 "Access denied when validating evaluator" error on every deploy with a custom evaluator
• Promote stability — promote ab-test waits for executionStatus === 'RUNNING' before stopping, avoiding 409 errors when promote runs immediately after resume; throws a clear error if the test never reaches RUNNING
• IAM — DescribeLogGroups — Split into its own statement with Resource: "*" (AWS requirement)
• E2E coverage — Added AB test reaches RUNNING status after deploy to ab-test-target-based.test.ts to catch IAM permission regressions post-deploy
• Status test — Fixed incorrect http-gateway resourceType assertion; replaced with ab-test + invocationUrl check
• Local e2e script — Added scripts/run-e2e-local.sh to replicate the GitHub Actions e2e workflow locally

IAM changes

Trust policy — added SourceAccount + SourceArn conditions to prevent confused deputy attacks:

"Condition": {
"StringEquals": { "aws:SourceAccount": "<accountId>" },
"ArnLike": { "aws:SourceArn": "arn:aws:bedrock-agentcore:*:<accountId>:ab-test/*" }
}

Permissions — 3-statement policy aligned with docs:

• AgentCoreResources — all bedrock-agentcore actions scoped to account with ResourceAccount condition (adds GetEvaluator, GetGatewayTarget, ListGatewayTargets, ListConfigurationBundleVersions)
• CloudWatchLogsDescribe — logs:DescribeLogGroups with Resource: "*" (required by AWS)
• CloudWatchLogs — log read/write actions scoped to evaluation and spans log groups

Running E2E tests locally

export E2E_ROLE_ARN=arn:aws:iam::<account>:role/<role>export E2E_SECRET_ARN=arn:aws:secretsmanager:<region>:<account>:secret:<name>
./scripts/run-e2e-local.sh e2e-tests/ab-test-target-based.test.ts

Test plan

• e2e-tests/ab-test-target-based.test.ts — all steps pass including RUNNING check
• e2e-tests/ab-test-config-bundle.test.ts — passes (RUNNING check deferred to follow-up once real config bundles are used)
• Unit tests src/cli/operations/deploy/__tests__/post-deploy-ab-tests.test.ts — 23 passing
• Unit tests src/cli/commands/pause/__tests__/promote.test.ts — 4 passing

[agentcore-cli-automation]

The new EvaluatorReadStatement is scoped to Object.values(deployedResources?.evaluators ?? {}), which only contains custom evaluators deployed by this project. This will leave the original 400 error in place whenever the AB test's online-eval config references evaluators that aren't in deployedResources.evaluators:

• Builtin evaluators (Builtin.Faithfulness, Builtin.GoalSuccessRate, etc.) — these are first-class evaluator references in OnlineEvalConfig.evaluators (see src/schema/schemas/primitives/online-eval-config.ts:23-24 and the handling in AgentCoreOnlineEvaluationConfig.ts:143-161). If the service validates evaluator ARNs uniformly using the execution role, Builtins will fail the same check the PR is trying to fix.
• External evaluator ARNs — OnlineEvalConfig.evaluators also accepts raw ARNs pointing at evaluators outside this project / another account. Those are never in deployedResources.evaluators either.

Separately, even for the custom-evaluator case, the policy grants GetEvaluator on every evaluator in the project rather than just the ones this specific AB test transitively references via its onlineEvalConfigArns. That's a least-privilege concern but not a functional blocker.

A couple of ways to address this:

1. Resolve evaluators transitively from the referenced online eval configs. For each online eval config referenced by testSpec.evaluationConfig, look up its evaluators list in projectSpec.onlineEvalConfigs, and for each entry:

   • Builtin.* → construct the builtin ARN (arn:${partition}:bedrock-agentcore:::evaluator/Builtin.*)
   • arn:... → use as-is
   • otherwise → look up in deployedResources.evaluators

   Pass that de-duplicated list as evaluatorArns. This fixes both the Builtin/external case and the over-broad scoping.

2. If tightening the scope is risky, at minimum add arn:${partition}:bedrock-agentcore:::evaluator/Builtin.* to the Resource list (and optionally arn:${partition}:bedrock-agentcore:${region}:${accountId}:evaluator/*) so users with builtin evaluators don't hit the same 400.

Could you also confirm whether you tested the fix with a Builtin evaluator in the online eval config, or only with a custom one? The test plan only mentions ab-test-target-based.test.ts / ab-test-config-bundle.test.ts — knowing which evaluator types those exercise would help.

[github-actions]

Coverage Report

Status	Category	Percentage	Covered / Total
🔵	Lines	43.11%	9016 / 20912
🔵	Statements	42.4%	9575 / 22582
🔵	Functions	39.96%	1555 / 3891
🔵	Branches	39.98%	5808 / 14527

Generated in workflow #2425 for commit 5b72ab3 by the Vitest Coverage Report Action

[jariy17]

Addressed in 44bc3f2. The evaluator ARN resolution is now transitive from the online eval configs the AB test references — handling all three cases:

• Custom evaluators — looked up from deployedResources.evaluators
• Builtin.* evaluators — ARN constructed as arn:${partition}:bedrock-agentcore:::evaluator/Builtin.*
• External ARN references — used as-is

The policy is now scoped to only the evaluators this specific AB test transitively references (not all project evaluators), addressing the least-privilege concern.

The e2e tests exercise a custom evaluator (ABTestEvaluator in ab-test-target-based.test.ts). We haven't explicitly tested with Builtin evaluators, but the ARN construction follows the same pattern used in the L3 CDK constructs (AgentCoreOnlineEvaluationConfig.ts:143-161).

[notgitika]

waitForRunningThenStop fixes the CLI promote race condition, but if there is a TUI path that calls updateABTest({ executionStatus: "STOPPED" }) directly, it has the same bug right?

[jariy17]

Summary

• IAM policy — Aligns the auto-created AB test execution role with the official docs, fixing a 400 "Access denied when validating evaluator" error on every deploy with a custom evaluator
• Promote stability — promote ab-test waits for executionStatus === 'RUNNING' before stopping, avoiding 409 errors when promote runs immediately after resume; throws a clear error if the test never reaches RUNNING
• IAM — DescribeLogGroups — Split into its own statement with Resource: "*" (AWS requirement)
• E2E coverage — Added AB test reaches RUNNING status after deploy to ab-test-target-based.test.ts to catch IAM permission regressions post-deploy
• Status test — Fixed incorrect http-gateway resourceType assertion; replaced with ab-test + invocationUrl check
• Local e2e script — Added scripts/run-e2e-local.sh to replicate the GitHub Actions e2e workflow locally

IAM changes

Trust policy — added SourceAccount + SourceArn conditions to prevent confused deputy attacks:

"Condition": {
"StringEquals": { "aws:SourceAccount": "<accountId>" },
"ArnLike": { "aws:SourceArn": "arn:aws:bedrock-agentcore:*:<accountId>:ab-test/*" }
}

Permissions — 3-statement policy aligned with docs:

• AgentCoreResources — all bedrock-agentcore actions scoped to account with ResourceAccount condition (adds GetEvaluator, GetGatewayTarget, ListGatewayTargets, ListConfigurationBundleVersions)
• CloudWatchLogsDescribe — logs:DescribeLogGroups with Resource: "*" (required by AWS)
• CloudWatchLogs — log read/write actions scoped to evaluation and spans log groups

Running E2E tests locally

export E2E_ROLE_ARN=arn:aws:iam::<account>:role/<role>export E2E_SECRET_ARN=arn:aws:secretsmanager:<region>:<account>:secret:<name>
./scripts/run-e2e-local.sh e2e-tests/ab-test-target-based.test.ts

Test plan

• e2e-tests/ab-test-target-based.test.ts — all steps pass including RUNNING check
• e2e-tests/ab-test-config-bundle.test.ts — passes (RUNNING check deferred to follow-up once real config bundles are used)
• Unit tests src/cli/operations/deploy/__tests__/post-deploy-ab-tests.test.ts — 23 passing
• Unit tests src/cli/commands/pause/__tests__/promote.test.ts — 4 passing

[agentcore-cli-automation]

The new EvaluatorReadStatement is scoped to Object.values(deployedResources?.evaluators ?? {}), which only contains custom evaluators deployed by this project. This will leave the original 400 error in place whenever the AB test's online-eval config references evaluators that aren't in deployedResources.evaluators:

• Builtin evaluators (Builtin.Faithfulness, Builtin.GoalSuccessRate, etc.) — these are first-class evaluator references in OnlineEvalConfig.evaluators (see src/schema/schemas/primitives/online-eval-config.ts:23-24 and the handling in AgentCoreOnlineEvaluationConfig.ts:143-161). If the service validates evaluator ARNs uniformly using the execution role, Builtins will fail the same check the PR is trying to fix.
• External evaluator ARNs — OnlineEvalConfig.evaluators also accepts raw ARNs pointing at evaluators outside this project / another account. Those are never in deployedResources.evaluators either.

Separately, even for the custom-evaluator case, the policy grants GetEvaluator on every evaluator in the project rather than just the ones this specific AB test transitively references via its onlineEvalConfigArns. That's a least-privilege concern but not a functional blocker.

A couple of ways to address this:

1. Resolve evaluators transitively from the referenced online eval configs. For each online eval config referenced by testSpec.evaluationConfig, look up its evaluators list in projectSpec.onlineEvalConfigs, and for each entry:

   • Builtin.* → construct the builtin ARN (arn:${partition}:bedrock-agentcore:::evaluator/Builtin.*)
   • arn:... → use as-is
   • otherwise → look up in deployedResources.evaluators

   Pass that de-duplicated list as evaluatorArns. This fixes both the Builtin/external case and the over-broad scoping.

2. If tightening the scope is risky, at minimum add arn:${partition}:bedrock-agentcore:::evaluator/Builtin.* to the Resource list (and optionally arn:${partition}:bedrock-agentcore:${region}:${accountId}:evaluator/*) so users with builtin evaluators don't hit the same 400.

Could you also confirm whether you tested the fix with a Builtin evaluator in the online eval config, or only with a custom one? The test plan only mentions ab-test-target-based.test.ts / ab-test-config-bundle.test.ts — knowing which evaluator types those exercise would help.

[github-actions]

Coverage Report

Status	Category	Percentage	Covered / Total
🔵	Lines	43.11%	9016 / 20912
🔵	Statements	42.4%	9575 / 22582
🔵	Functions	39.96%	1555 / 3891
🔵	Branches	39.98%	5808 / 14527

Generated in workflow #2425 for commit 5b72ab3 by the Vitest Coverage Report Action

[jariy17]

Addressed in 44bc3f2. The evaluator ARN resolution is now transitive from the online eval configs the AB test references — handling all three cases:

• Custom evaluators — looked up from deployedResources.evaluators
• Builtin.* evaluators — ARN constructed as arn:${partition}:bedrock-agentcore:::evaluator/Builtin.*
• External ARN references — used as-is

The policy is now scoped to only the evaluators this specific AB test transitively references (not all project evaluators), addressing the least-privilege concern.

The e2e tests exercise a custom evaluator (ABTestEvaluator in ab-test-target-based.test.ts). We haven't explicitly tested with Builtin evaluators, but the ARN construction follows the same pattern used in the L3 CDK constructs (AgentCoreOnlineEvaluationConfig.ts:143-161).

[notgitika]

waitForRunningThenStop fixes the CLI promote race condition, but if there is a TUI path that calls updateABTest({ executionStatus: "STOPPED" }) directly, it has the same bug right?

[jariy17]

Summary

• IAM policy — Aligns the auto-created AB test execution role with the official docs, fixing a 400 "Access denied when validating evaluator" error on every deploy with a custom evaluator
• Promote stability — promote ab-test waits for executionStatus === 'RUNNING' before stopping, avoiding 409 errors when promote runs immediately after resume; throws a clear error if the test never reaches RUNNING
• IAM — DescribeLogGroups — Split into its own statement with Resource: "*" (AWS requirement)
• E2E coverage — Added AB test reaches RUNNING status after deploy to ab-test-target-based.test.ts to catch IAM permission regressions post-deploy
• Status test — Fixed incorrect http-gateway resourceType assertion; replaced with ab-test + invocationUrl check
• Local e2e script — Added scripts/run-e2e-local.sh to replicate the GitHub Actions e2e workflow locally

IAM changes

Trust policy — added SourceAccount + SourceArn conditions to prevent confused deputy attacks:

"Condition": {
"StringEquals": { "aws:SourceAccount": "<accountId>" },
"ArnLike": { "aws:SourceArn": "arn:aws:bedrock-agentcore:*:<accountId>:ab-test/*" }
}

Permissions — 3-statement policy aligned with docs:

• AgentCoreResources — all bedrock-agentcore actions scoped to account with ResourceAccount condition (adds GetEvaluator, GetGatewayTarget, ListGatewayTargets, ListConfigurationBundleVersions)
• CloudWatchLogsDescribe — logs:DescribeLogGroups with Resource: "*" (required by AWS)
• CloudWatchLogs — log read/write actions scoped to evaluation and spans log groups

Running E2E tests locally

export E2E_ROLE_ARN=arn:aws:iam::<account>:role/<role>export E2E_SECRET_ARN=arn:aws:secretsmanager:<region>:<account>:secret:<name>
./scripts/run-e2e-local.sh e2e-tests/ab-test-target-based.test.ts

Test plan

• e2e-tests/ab-test-target-based.test.ts — all steps pass including RUNNING check
• e2e-tests/ab-test-config-bundle.test.ts — passes (RUNNING check deferred to follow-up once real config bundles are used)
• Unit tests src/cli/operations/deploy/__tests__/post-deploy-ab-tests.test.ts — 23 passing
• Unit tests src/cli/commands/pause/__tests__/promote.test.ts — 4 passing

[agentcore-cli-automation]

The new EvaluatorReadStatement is scoped to Object.values(deployedResources?.evaluators ?? {}), which only contains custom evaluators deployed by this project. This will leave the original 400 error in place whenever the AB test's online-eval config references evaluators that aren't in deployedResources.evaluators:

• Builtin evaluators (Builtin.Faithfulness, Builtin.GoalSuccessRate, etc.) — these are first-class evaluator references in OnlineEvalConfig.evaluators (see src/schema/schemas/primitives/online-eval-config.ts:23-24 and the handling in AgentCoreOnlineEvaluationConfig.ts:143-161). If the service validates evaluator ARNs uniformly using the execution role, Builtins will fail the same check the PR is trying to fix.
• External evaluator ARNs — OnlineEvalConfig.evaluators also accepts raw ARNs pointing at evaluators outside this project / another account. Those are never in deployedResources.evaluators either.

Separately, even for the custom-evaluator case, the policy grants GetEvaluator on every evaluator in the project rather than just the ones this specific AB test transitively references via its onlineEvalConfigArns. That's a least-privilege concern but not a functional blocker.

A couple of ways to address this:

1. Resolve evaluators transitively from the referenced online eval configs. For each online eval config referenced by testSpec.evaluationConfig, look up its evaluators list in projectSpec.onlineEvalConfigs, and for each entry:

   • Builtin.* → construct the builtin ARN (arn:${partition}:bedrock-agentcore:::evaluator/Builtin.*)
   • arn:... → use as-is
   • otherwise → look up in deployedResources.evaluators

   Pass that de-duplicated list as evaluatorArns. This fixes both the Builtin/external case and the over-broad scoping.

2. If tightening the scope is risky, at minimum add arn:${partition}:bedrock-agentcore:::evaluator/Builtin.* to the Resource list (and optionally arn:${partition}:bedrock-agentcore:${region}:${accountId}:evaluator/*) so users with builtin evaluators don't hit the same 400.

Could you also confirm whether you tested the fix with a Builtin evaluator in the online eval config, or only with a custom one? The test plan only mentions ab-test-target-based.test.ts / ab-test-config-bundle.test.ts — knowing which evaluator types those exercise would help.

[github-actions]

Coverage Report

Status	Category	Percentage	Covered / Total
🔵	Lines	43.11%	9016 / 20912
🔵	Statements	42.4%	9575 / 22582
🔵	Functions	39.96%	1555 / 3891
🔵	Branches	39.98%	5808 / 14527

Generated in workflow #2425 for commit 5b72ab3 by the Vitest Coverage Report Action

[jariy17]

Addressed in 44bc3f2. The evaluator ARN resolution is now transitive from the online eval configs the AB test references — handling all three cases:

• Custom evaluators — looked up from deployedResources.evaluators
• Builtin.* evaluators — ARN constructed as arn:${partition}:bedrock-agentcore:::evaluator/Builtin.*
• External ARN references — used as-is

The policy is now scoped to only the evaluators this specific AB test transitively references (not all project evaluators), addressing the least-privilege concern.

The e2e tests exercise a custom evaluator (ABTestEvaluator in ab-test-target-based.test.ts). We haven't explicitly tested with Builtin evaluators, but the ARN construction follows the same pattern used in the L3 CDK constructs (AgentCoreOnlineEvaluationConfig.ts:143-161).

[notgitika]

waitForRunningThenStop fixes the CLI promote race condition, but if there is a TUI path that calls updateABTest({ executionStatus: "STOPPED" }) directly, it has the same bug right?

[jariy17]

Summary

• IAM policy — Aligns the auto-created AB test execution role with the official docs, fixing a 400 "Access denied when validating evaluator" error on every deploy with a custom evaluator
• Promote stability — promote ab-test waits for executionStatus === 'RUNNING' before stopping, avoiding 409 errors when promote runs immediately after resume; throws a clear error if the test never reaches RUNNING
• IAM — DescribeLogGroups — Split into its own statement with Resource: "*" (AWS requirement)
• E2E coverage — Added AB test reaches RUNNING status after deploy to ab-test-target-based.test.ts to catch IAM permission regressions post-deploy
• Status test — Fixed incorrect http-gateway resourceType assertion; replaced with ab-test + invocationUrl check
• Local e2e script — Added scripts/run-e2e-local.sh to replicate the GitHub Actions e2e workflow locally

IAM changes

Trust policy — added SourceAccount + SourceArn conditions to prevent confused deputy attacks:

"Condition": {
"StringEquals": { "aws:SourceAccount": "<accountId>" },
"ArnLike": { "aws:SourceArn": "arn:aws:bedrock-agentcore:*:<accountId>:ab-test/*" }
}

Permissions — 3-statement policy aligned with docs:

• AgentCoreResources — all bedrock-agentcore actions scoped to account with ResourceAccount condition (adds GetEvaluator, GetGatewayTarget, ListGatewayTargets, ListConfigurationBundleVersions)
• CloudWatchLogsDescribe — logs:DescribeLogGroups with Resource: "*" (required by AWS)
• CloudWatchLogs — log read/write actions scoped to evaluation and spans log groups

Running E2E tests locally

export E2E_ROLE_ARN=arn:aws:iam::<account>:role/<role>export E2E_SECRET_ARN=arn:aws:secretsmanager:<region>:<account>:secret:<name>
./scripts/run-e2e-local.sh e2e-tests/ab-test-target-based.test.ts

Test plan

• e2e-tests/ab-test-target-based.test.ts — all steps pass including RUNNING check
• e2e-tests/ab-test-config-bundle.test.ts — passes (RUNNING check deferred to follow-up once real config bundles are used)
• Unit tests src/cli/operations/deploy/__tests__/post-deploy-ab-tests.test.ts — 23 passing
• Unit tests src/cli/commands/pause/__tests__/promote.test.ts — 4 passing

[agentcore-cli-automation]

The new EvaluatorReadStatement is scoped to Object.values(deployedResources?.evaluators ?? {}), which only contains custom evaluators deployed by this project. This will leave the original 400 error in place whenever the AB test's online-eval config references evaluators that aren't in deployedResources.evaluators:

• Builtin evaluators (Builtin.Faithfulness, Builtin.GoalSuccessRate, etc.) — these are first-class evaluator references in OnlineEvalConfig.evaluators (see src/schema/schemas/primitives/online-eval-config.ts:23-24 and the handling in AgentCoreOnlineEvaluationConfig.ts:143-161). If the service validates evaluator ARNs uniformly using the execution role, Builtins will fail the same check the PR is trying to fix.
• External evaluator ARNs — OnlineEvalConfig.evaluators also accepts raw ARNs pointing at evaluators outside this project / another account. Those are never in deployedResources.evaluators either.

Separately, even for the custom-evaluator case, the policy grants GetEvaluator on every evaluator in the project rather than just the ones this specific AB test transitively references via its onlineEvalConfigArns. That's a least-privilege concern but not a functional blocker.

A couple of ways to address this:

1. Resolve evaluators transitively from the referenced online eval configs. For each online eval config referenced by testSpec.evaluationConfig, look up its evaluators list in projectSpec.onlineEvalConfigs, and for each entry:

   • Builtin.* → construct the builtin ARN (arn:${partition}:bedrock-agentcore:::evaluator/Builtin.*)
   • arn:... → use as-is
   • otherwise → look up in deployedResources.evaluators

   Pass that de-duplicated list as evaluatorArns. This fixes both the Builtin/external case and the over-broad scoping.

2. If tightening the scope is risky, at minimum add arn:${partition}:bedrock-agentcore:::evaluator/Builtin.* to the Resource list (and optionally arn:${partition}:bedrock-agentcore:${region}:${accountId}:evaluator/*) so users with builtin evaluators don't hit the same 400.

Could you also confirm whether you tested the fix with a Builtin evaluator in the online eval config, or only with a custom one? The test plan only mentions ab-test-target-based.test.ts / ab-test-config-bundle.test.ts — knowing which evaluator types those exercise would help.

[github-actions]

Coverage Report

Status	Category	Percentage	Covered / Total
🔵	Lines	43.11%	9016 / 20912
🔵	Statements	42.4%	9575 / 22582
🔵	Functions	39.96%	1555 / 3891
🔵	Branches	39.98%	5808 / 14527

Generated in workflow #2425 for commit 5b72ab3 by the Vitest Coverage Report Action

[jariy17]

Addressed in 44bc3f2. The evaluator ARN resolution is now transitive from the online eval configs the AB test references — handling all three cases:

• Custom evaluators — looked up from deployedResources.evaluators
• Builtin.* evaluators — ARN constructed as arn:${partition}:bedrock-agentcore:::evaluator/Builtin.*
• External ARN references — used as-is

The policy is now scoped to only the evaluators this specific AB test transitively references (not all project evaluators), addressing the least-privilege concern.

The e2e tests exercise a custom evaluator (ABTestEvaluator in ab-test-target-based.test.ts). We haven't explicitly tested with Builtin evaluators, but the ARN construction follows the same pattern used in the L3 CDK constructs (AgentCoreOnlineEvaluationConfig.ts:143-161).

[notgitika]

waitForRunningThenStop fixes the CLI promote race condition, but if there is a TUI path that calls updateABTest({ executionStatus: "STOPPED" }) directly, it has the same bug right?

[jariy17]

Summary

• IAM policy — Aligns the auto-created AB test execution role with the official docs, fixing a 400 "Access denied when validating evaluator" error on every deploy with a custom evaluator
• Promote stability — promote ab-test waits for executionStatus === 'RUNNING' before stopping, avoiding 409 errors when promote runs immediately after resume; throws a clear error if the test never reaches RUNNING
• IAM — DescribeLogGroups — Split into its own statement with Resource: "*" (AWS requirement)
• E2E coverage — Added AB test reaches RUNNING status after deploy to ab-test-target-based.test.ts to catch IAM permission regressions post-deploy
• Status test — Fixed incorrect http-gateway resourceType assertion; replaced with ab-test + invocationUrl check
• Local e2e script — Added scripts/run-e2e-local.sh to replicate the GitHub Actions e2e workflow locally

IAM changes

Trust policy — added SourceAccount + SourceArn conditions to prevent confused deputy attacks:

"Condition": {
"StringEquals": { "aws:SourceAccount": "<accountId>" },
"ArnLike": { "aws:SourceArn": "arn:aws:bedrock-agentcore:*:<accountId>:ab-test/*" }
}

Permissions — 3-statement policy aligned with docs:

• AgentCoreResources — all bedrock-agentcore actions scoped to account with ResourceAccount condition (adds GetEvaluator, GetGatewayTarget, ListGatewayTargets, ListConfigurationBundleVersions)
• CloudWatchLogsDescribe — logs:DescribeLogGroups with Resource: "*" (required by AWS)
• CloudWatchLogs — log read/write actions scoped to evaluation and spans log groups

Running E2E tests locally

export E2E_ROLE_ARN=arn:aws:iam::<account>:role/<role>export E2E_SECRET_ARN=arn:aws:secretsmanager:<region>:<account>:secret:<name>
./scripts/run-e2e-local.sh e2e-tests/ab-test-target-based.test.ts

Test plan

• e2e-tests/ab-test-target-based.test.ts — all steps pass including RUNNING check
• e2e-tests/ab-test-config-bundle.test.ts — passes (RUNNING check deferred to follow-up once real config bundles are used)
• Unit tests src/cli/operations/deploy/__tests__/post-deploy-ab-tests.test.ts — 23 passing
• Unit tests src/cli/commands/pause/__tests__/promote.test.ts — 4 passing

[agentcore-cli-automation]

The new EvaluatorReadStatement is scoped to Object.values(deployedResources?.evaluators ?? {}), which only contains custom evaluators deployed by this project. This will leave the original 400 error in place whenever the AB test's online-eval config references evaluators that aren't in deployedResources.evaluators:

• Builtin evaluators (Builtin.Faithfulness, Builtin.GoalSuccessRate, etc.) — these are first-class evaluator references in OnlineEvalConfig.evaluators (see src/schema/schemas/primitives/online-eval-config.ts:23-24 and the handling in AgentCoreOnlineEvaluationConfig.ts:143-161). If the service validates evaluator ARNs uniformly using the execution role, Builtins will fail the same check the PR is trying to fix.
• External evaluator ARNs — OnlineEvalConfig.evaluators also accepts raw ARNs pointing at evaluators outside this project / another account. Those are never in deployedResources.evaluators either.

Separately, even for the custom-evaluator case, the policy grants GetEvaluator on every evaluator in the project rather than just the ones this specific AB test transitively references via its onlineEvalConfigArns. That's a least-privilege concern but not a functional blocker.

A couple of ways to address this:

1. Resolve evaluators transitively from the referenced online eval configs. For each online eval config referenced by testSpec.evaluationConfig, look up its evaluators list in projectSpec.onlineEvalConfigs, and for each entry:

   • Builtin.* → construct the builtin ARN (arn:${partition}:bedrock-agentcore:::evaluator/Builtin.*)
   • arn:... → use as-is
   • otherwise → look up in deployedResources.evaluators

   Pass that de-duplicated list as evaluatorArns. This fixes both the Builtin/external case and the over-broad scoping.

2. If tightening the scope is risky, at minimum add arn:${partition}:bedrock-agentcore:::evaluator/Builtin.* to the Resource list (and optionally arn:${partition}:bedrock-agentcore:${region}:${accountId}:evaluator/*) so users with builtin evaluators don't hit the same 400.

Could you also confirm whether you tested the fix with a Builtin evaluator in the online eval config, or only with a custom one? The test plan only mentions ab-test-target-based.test.ts / ab-test-config-bundle.test.ts — knowing which evaluator types those exercise would help.

[github-actions]

Coverage Report

Status	Category	Percentage	Covered / Total
🔵	Lines	43.11%	9016 / 20912
🔵	Statements	42.4%	9575 / 22582
🔵	Functions	39.96%	1555 / 3891
🔵	Branches	39.98%	5808 / 14527

Generated in workflow #2425 for commit 5b72ab3 by the Vitest Coverage Report Action

[jariy17]

Addressed in 44bc3f2. The evaluator ARN resolution is now transitive from the online eval configs the AB test references — handling all three cases:

• Custom evaluators — looked up from deployedResources.evaluators
• Builtin.* evaluators — ARN constructed as arn:${partition}:bedrock-agentcore:::evaluator/Builtin.*
• External ARN references — used as-is

The policy is now scoped to only the evaluators this specific AB test transitively references (not all project evaluators), addressing the least-privilege concern.

The e2e tests exercise a custom evaluator (ABTestEvaluator in ab-test-target-based.test.ts). We haven't explicitly tested with Builtin evaluators, but the ARN construction follows the same pattern used in the L3 CDK constructs (AgentCoreOnlineEvaluationConfig.ts:143-161).

[notgitika]

waitForRunningThenStop fixes the CLI promote race condition, but if there is a TUI path that calls updateABTest({ executionStatus: "STOPPED" }) directly, it has the same bug right?