Uh oh!
There was an error while loading. Please reload this page.
fix: correct AB test execution role IAM policy and promote stability - #1126
Merged
Merged
Conversation
…1120) * fix: add GetEvaluator permission to AB test execution role The AB test API validates evaluator ARNs server-side using the execution role. The auto-created role policy was missing bedrock-agentcore:GetEvaluator, causing a 400 "Access denied when validating evaluator" error on every deploy that included a target-based AB test with a custom evaluator. * fix: correct status assertion in target-based AB test e2e HTTP gateways are not surfaced as top-level resources in agentcore status — they are only used internally to build AB test invocation URLs. The test was asserting on resourceType 'http-gateway' which never appears; fix it to assert on the 'ab-test' resource instead. * fix: add invocationUrl to status resource type cast in e2e test The resource cast was missing invocationUrl, causing a TypeScript error when asserting the AB test's gateway invocation URL was present. * fix: improve promote error message and add local e2e run script - Include stdout in promote failure message so the JSON error is visible - Add scripts/run-e2e-local.sh to replicate the GitHub Actions e2e workflow locally * fix: retry promote stop on 409 UPDATING status When promote is called immediately after resume, the AB test may still be in UPDATING state and reject the STOPPED transition with a 409. Retry up to 6 times with a 10s delay to wait for the transition to complete. * fix: wait for AB test to leave UPDATING state before promoting Poll getABTest until executionStatus is no longer UPDATING before attempting to stop. The service rejects updates with 409 while a state transition is in progress (e.g. after resume). * fix: wait for RUNNING before stopping AB test in promote Poll getABTest until executionStatus === 'RUNNING' before issuing the STOPPED transition. The service 409s if the test is still UPDATING (e.g. transitioning from a prior resume). * fix: resolve evaluator ARNs transitively from AB test online eval configs Previously GetEvaluator was scoped to all project evaluators. Now it's scoped to only the evaluators referenced by the specific online eval configs this AB test uses, handling all three cases: - Custom evaluators: looked up from deployedResources.evaluators - Builtin.* evaluators: ARN constructed from the builtin ID - External ARN references: used as-is * fix: align AB test execution role policy with official docs Replace the hand-rolled per-resource policy with the canonical policy from https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/ab-testing-prereqs.html: - Trust policy: add SourceAccount + SourceArn conditions - Permissions: single AgentCoreResources statement scoped to arn:aws:bedrock-agentcore:*:${accountId}:* with ResourceAccount condition - Add missing actions: GetGatewayTarget, ListGatewayTargets, ListConfigurationBundleVersions - CloudWatch logs scoped to account via PrincipalAccount pattern - Remove per-evaluator/per-resource ARN tracking (no longer needed) * test: update IAM role unit tests to match new policy structure * fix: throw error if AB test never reaches RUNNING before promote * test: add unit tests for promote polling logic and extract to promote-utils Extract waitForRunningThenStop into promote-utils.ts so it can be unit tested without pulling in React/ink. Add 4 tests covering: immediate RUNNING, polling until RUNNING, timeout throws, and error message content. * fix: split DescribeLogGroups into wildcard resource statement * test: verify AB test reaches RUNNING status after deploy in both e2e suites * fix: remove console.log statements that leaked ARNs and fix bundle ARN/version format in config-bundle e2e test * test: remove second deploy and RUNNING check from config-bundle e2e (follow-up with real bundles) --------- Co-authored-by: Local E2E <ci@local>
agentcore-cli-automation
commented
May 5, 2026
Reviewed the cherry-pick. All the serious issues I'd raise have already been flagged in the existing discussion:
The cherry-pick onto |
Contributor
Coverage Report
|
Uh oh!
There was an error while loading. Please reload this page.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cherry-pick of #1120 onto
preview.Summary
"Access denied when validating evaluator"error on deploypromote ab-testwaits forexecutionStatus === 'RUNNING'before stopping, avoiding 409 errors when promote runs immediately after resumeResource: "*"(AWS requirement)AB test reaches RUNNING status after deploytoab-test-target-based.test.tshttp-gatewayresourceType assertionscripts/run-e2e-local.shSee #1120 for full details.