Uh oh!
There was an error while loading. Please reload this page.
feat(eval): read-only ab-test commands + TUI - #2102
Merged
Merged
Conversation
Contributor
Claude Security Review: no high-confidence findings. (run) |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@## refactor #2102 +/- ##
============================================
- Coverage 97.33% 97.30% -0.04%
============================================
Files 417 428 +11 Lines 25250 25546 +296 ============================================
+ Hits 24578 24858 +280 - Misses 672 688 +16 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Add `agentcore eval ab-test get` and `list`, modeled on batch-evaluation. A/B tests are imperative jobs on the data-plane client, so this wires the existing @aws-sdk/client-bedrock-agentcore GetABTestCommand / ListABTestsCommand through EvalClient — no SigV4, no new module. - get: --id only; GetABTest returns per-evaluator metrics inline (no CloudWatch) - list: --next-token / --max-results passthrough (service supports no filters) - TestCoreClient: getABTest/listABTests mocks + setters - command-flow tests TUI screens and mutating commands deferred to later phases.
jariy17force-pushed
the
feat/eval-ab-test-readonly
branch
from
August 25, 2026 20:04
6228f89 to
520f0beCompareContributor
Claude Security Review: no high-confidence findings. (run) |
jariy17
marked this pull request as ready for review
August 25, 2026 20:06
Interactive screens for `agentcore eval ab-test`, modeled on the runtime TUI: - AbTestScreen menu (RouterScreen) - AbTestListScreen → AbTestPicker (wraps PaginatedTablePicker, listABTests) - AbTestGetScreen detail hub (ResourceDetailScreen) + AbTestGetJsonScreen - 5 routes wired in Root.tsx Screen tests via renderScreen mirror runtime.screen.test.tsx. No CLI/core changes; reuses the getABTest/listABTests methods + mocks from the read-only commit.
Contributor
Claude Security Review: no high-confidence findings. (run) |
notgitika
previously approved these changes
Aug 25, 2026
Comment on lines
+29
to
+30
| status: summary.status ?? "-", | ||
| executionStatus: summary.executionStatus ?? "-", |
Contributor
There was a problem hiding this comment.
discussed offline and we see that this is the output from the API. i would imagine as customer would be confused seeing these 2 fields in the data table in TUI. we can try to unify it in the future maybe
| ], | ||
| }, | ||
| ...overrides, | ||
| } as GetABTestResponse; |
Contributor
There was a problem hiding this comment.
is the reason we are not using satisfies that we have treatmentname and not variantname? I see that in the fixture
ContributorAuthor
There was a problem hiding this comment.
Good catch — switched to satisfies GetABTestResponse in c2e0c18. It immediately surfaced that the stub had treatmentName (the real metric field is variantName, matching the recorded fixture) and was missing Variant.variantConfiguration. Both fixed; the cast was hiding them.
Comment on lines
+397
to
+398
| nextToken: string | undefined, | ||
| maxResults: number | undefined, |
Contributor
There was a problem hiding this comment.
nice that we have these as just passthroughs :)
Addresses review: swap the `as GetABTestResponse` cast for `satisfies`, which surfaced two wrong fields in the stub — the metric field is variantName (not treatmentName) and Variant requires variantConfiguration. Fill the required response fields so the stub type-checks.
Contributor
Claude Security Review: no high-confidence findings. (run) |
notgitika
previously approved these changes
Aug 25, 2026
pause/resume/stop set executionStatus via UpdateABTestCommand (same shape as online-eval pause/resume, on the data-plane client); delete uses DeleteABTestCommand. All four are --id-only leaf handlers registered on the ab-test router. stop carries a TODO for the (never-executed) gateway-rule promotion suggestion. Command-flow tests cover the hierarchy, each transition's executionStatus, delete, and the required --id.
jariy17
marked this pull request as draft
August 25, 2026 23:24
Contributor
Claude Security Review: no high-confidence findings. (run) |
Add .supportedTuiCommands("get", "list") to the ab-test router, matching
online-eval. Without it the interactive menu listed pause/resume/stop/delete,
which have no screen routes and dead-ended on the help screen. Add a menu
screen test that fails without the fix.Contributor
Claude Security Review: no high-confidence findings. (run) |
AlexanderRichey
approved these changes
Aug 25, 2026
jariy17
marked this pull request as ready for review
August 26, 2026 14:21
Uh oh!
There was an error while loading. Please reload this page.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the full read-only + write
agentcore eval ab-testcommand family, plus the interactive TUI. Five commits.Command + TUI structure
How it works
A/B tests are imperative jobs (like
batch-evaluation), not project resources. All commands are one-liners through the data-plane SDK client@aws-sdk/client-bedrock-agentcore:get/list→GetABTestCommand/ListABTestsCommand.getreturns per-evaluator metrics inline (no CloudWatch).pause/resume/stop→UpdateABTestCommand({ abTestId, executionStatus })— the same shapeonline-evalpause/resume uses.delete→DeleteABTestCommand.runtime. Write commands are CLI-only and kept out of the interactive menu via.supportedTuiCommands("get","list")(matchesonline-eval).Known / deferred
stopgateway-rule suggestion — after stopping, the doc calls for printing a suggested (never executed)update-gateway-ruleto promote the winner. Left as a// TODOthis PR; tracked as a follow-up.--jsonerror path — failures print plain-textError: …to stderr, not a JSON error object. Shared-renderer issue, not ab-test-specific; follow-up.online-eval.Files
handlers/eval/ab-test/{get,list,pause,resume,stop,delete}/index.tsx,index.tsxhandlers/eval/ab-test/{screen,list/screen,get/screen}.tsx,components/AbTestPicker.tsx, 5 routes inRoot.tsxcore/eval.tsx(getABTest/listABTests/setABTestExecutionStatus/deleteABTest),handlers/eval/types.tsx,testing/TestCoreClient.tsxTests
ab-test.fixture.test.tsx, likeharness): real router → CoreClient → recorded SDK →matchGolden. get / list / not-found.ab-test.screen.test.tsx, likeruntime): menu (read-only), picker, hub, json, retry.ab-test.write.test.tsx, likebatch-evaluation): hierarchy = get/list/pause/resume/stop/delete; each transition'sexecutionStatus; delete; required--id.Bug bash — all green
bun test src/handlers/eval src/components→ 338+ pass / 0 fail;bun run typecheck→ 0 errors;oxlint→ clean./review-pr+ a live non-destructive bug-bash on the exploratory account (us-west-2): all 6 commands dispatch correctly; write verbs return a clean not-found on a bogus id (no mutation);--idguard fires pre-network. One real bug found & fixed in this PR: the TUI menu listed write commands (dead-end) — now.supportedTuiCommands.Follow-ups (separate PRs)
stop's gateway-rule promotion suggestion--jsonerror output (shared renderer)projectSchemas/ab-test.ts(contradicts the job model)