feat: add ground truth reference inputs for on-demand evaluation by jariy17 · Pull Request #732 · aws/agentcore-cli

jariy17 · 2026-03-30T19:48:17Z

Summary

Adds ground truth reference inputs to on-demand evaluation (run eval), allowing users to specify expected agent behavior for more precise evaluation.

New CLI flags

-A, --assertion <text...> — Assertion the agent should satisfy (repeatable)
--expected-trajectory <names> — Expected tool calls in order (comma-separated)
--expected-response <text> — Expected agent response text

TUI changes

New Ground Truth wizard step in run eval flow (appears when exactly 1 session is selected)
Evaluator creation screen now shows reference input placeholders ({assertions}, {expected_tool_trajectory}, {actual_tool_trajectory}, {expected_response})
Updated default evaluator instructions to include GT placeholders

Implementation

evaluationReferenceInputs passed through to the evaluate() API call
Results include referenceInputs summary in both CLI and TUI output
Note: The second commit (temp: use middleware...) is a temporary workaround — the SDK does not yet include evaluationReferenceInputs in its model, so we inject it via Smithy middleware. Remove when SDK is updated.

Test plan

npx tsc --noEmit — clean
npm run lint — 0 errors
5 new unit tests for GT reference inputs (all pass)
Existing 33 run-eval tests continue to pass

github-actions · 2026-03-30T19:49:02Z

Package Tarball

aws-agentcore-0.4.0.tgz

How to install

npm install https://github.com/aws/agentcore-cli/releases/download/pr-732-tarball/aws-agentcore-0.4.0.tgz

github-actions · 2026-03-30T19:54:57Z

Coverage Report

Status	Category	Percentage	Covered / Total
🔵	Lines	45.98%	6575 / 14297
🔵	Statements	45.56%	6986 / 15333
🔵	Functions	44.68%	1177 / 2634
🔵	Branches	46.18%	4362 / 9444

Generated in workflow #1538 for commit d5e84ae by the Vitest Coverage Report Action

Bump @aws-sdk/client-bedrock-agentcore and @aws-sdk/client-bedrock-agentcore-control from ^3.893.0 to ^3.1020.0. The SDK now includes EvaluationReferenceInput in its model, so we pass it directly to EvaluateCommand instead of injecting via middleware.

jariy17 · 2026-03-30T20:42:49Z

Testing Results

Verified all new GT features against a deployed runtime (TempAgent in EvalCheck project):

CLI flags tested

-A, --assertion — single and multiple assertions with Builtin.GoalSuccessRate
--expected-trajectory — comma-separated tool names with Builtin.TrajectoryExactOrderMatch
--expected-response — with and without explicit -t trace ID, using Builtin.Correctness
All three combined in a single run across multiple evaluators
Baseline run (no GT inputs) — confirms referenceInputs is absent from output

Custom evaluators tested

MyEvaluatorTraceGT (TRACE level) — {expected_response} placeholder populated correctly
CustomEvaluatorSessionGT (SESSION level) — {assertions}, {expected_tool_trajectory}, {actual_tool_trajectory} placeholders populated correctly

SDK upgrade verified

@aws-sdk/client-bedrock-agentcore@^3.1020.0 — evaluationReferenceInputs passed natively via EvaluateCommand (no middleware workaround needed)

Other checks

npx tsc --noEmit — clean
npm run lint — 0 errors
All 38 unit tests pass (including 5 new GT tests)
TUI run eval wizard shows Ground Truth step when exactly 1 session is selected

Hweinstock

Some nits and question on service behavior. Otherwise lgtm.

Hweinstock

Slightly confused on the customer experience, but thats likely because I'm new to evals. The changes themselves lgtm.

Cover 13 PRs merged to main since the initial docs audit (March 28): - import subcommands: runtime, memory, evaluator, online-eval (#763, #780) - invoke/dev exec mode with --exec and --timeout (#750) - code-based evaluator support with --type, --lambda-arn, --timeout (#739) - ground truth eval inputs: --assertion, --expected-trajectory, --expected-response (#732) - memory record streaming: --data-stream-arn, --stream-content-level (#531, #534) - create --skip-install flag (#782) - fetch access --identity-name option (#774)

jariy17 · 2026-03-30T19:48:17Z

Summary

Adds ground truth reference inputs to on-demand evaluation (run eval), allowing users to specify expected agent behavior for more precise evaluation.

New CLI flags

-A, --assertion <text...> — Assertion the agent should satisfy (repeatable)
--expected-trajectory <names> — Expected tool calls in order (comma-separated)
--expected-response <text> — Expected agent response text

TUI changes

New Ground Truth wizard step in run eval flow (appears when exactly 1 session is selected)
Evaluator creation screen now shows reference input placeholders ({assertions}, {expected_tool_trajectory}, {actual_tool_trajectory}, {expected_response})
Updated default evaluator instructions to include GT placeholders

Implementation

evaluationReferenceInputs passed through to the evaluate() API call
Results include referenceInputs summary in both CLI and TUI output
Note: The second commit (temp: use middleware...) is a temporary workaround — the SDK does not yet include evaluationReferenceInputs in its model, so we inject it via Smithy middleware. Remove when SDK is updated.

Test plan

npx tsc --noEmit — clean
npm run lint — 0 errors
5 new unit tests for GT reference inputs (all pass)
Existing 33 run-eval tests continue to pass

github-actions · 2026-03-30T19:49:02Z

Package Tarball

aws-agentcore-0.4.0.tgz

How to install

npm install https://github.com/aws/agentcore-cli/releases/download/pr-732-tarball/aws-agentcore-0.4.0.tgz

github-actions · 2026-03-30T19:54:57Z

Coverage Report

Status	Category	Percentage	Covered / Total
🔵	Lines	45.98%	6575 / 14297
🔵	Statements	45.56%	6986 / 15333
🔵	Functions	44.68%	1177 / 2634
🔵	Branches	46.18%	4362 / 9444

Generated in workflow #1538 for commit d5e84ae by the Vitest Coverage Report Action

Bump @aws-sdk/client-bedrock-agentcore and @aws-sdk/client-bedrock-agentcore-control from ^3.893.0 to ^3.1020.0. The SDK now includes EvaluationReferenceInput in its model, so we pass it directly to EvaluateCommand instead of injecting via middleware.

jariy17 · 2026-03-30T20:42:49Z

Testing Results

Verified all new GT features against a deployed runtime (TempAgent in EvalCheck project):

CLI flags tested

-A, --assertion — single and multiple assertions with Builtin.GoalSuccessRate
--expected-trajectory — comma-separated tool names with Builtin.TrajectoryExactOrderMatch
--expected-response — with and without explicit -t trace ID, using Builtin.Correctness
All three combined in a single run across multiple evaluators
Baseline run (no GT inputs) — confirms referenceInputs is absent from output

Custom evaluators tested

MyEvaluatorTraceGT (TRACE level) — {expected_response} placeholder populated correctly
CustomEvaluatorSessionGT (SESSION level) — {assertions}, {expected_tool_trajectory}, {actual_tool_trajectory} placeholders populated correctly

SDK upgrade verified

@aws-sdk/client-bedrock-agentcore@^3.1020.0 — evaluationReferenceInputs passed natively via EvaluateCommand (no middleware workaround needed)

Other checks

npx tsc --noEmit — clean
npm run lint — 0 errors
All 38 unit tests pass (including 5 new GT tests)
TUI run eval wizard shows Ground Truth step when exactly 1 session is selected

Hweinstock

Some nits and question on service behavior. Otherwise lgtm.

Hweinstock

Slightly confused on the customer experience, but thats likely because I'm new to evals. The changes themselves lgtm.

Cover 13 PRs merged to main since the initial docs audit (March 28): - import subcommands: runtime, memory, evaluator, online-eval (#763, #780) - invoke/dev exec mode with --exec and --timeout (#750) - code-based evaluator support with --type, --lambda-arn, --timeout (#739) - ground truth eval inputs: --assertion, --expected-trajectory, --expected-response (#732) - memory record streaming: --data-stream-arn, --stream-content-level (#531, #534) - create --skip-install flag (#782) - fetch access --identity-name option (#774)

jariy17 · 2026-03-30T19:48:17Z

Summary

Adds ground truth reference inputs to on-demand evaluation (run eval), allowing users to specify expected agent behavior for more precise evaluation.

New CLI flags

-A, --assertion <text...> — Assertion the agent should satisfy (repeatable)
--expected-trajectory <names> — Expected tool calls in order (comma-separated)
--expected-response <text> — Expected agent response text

TUI changes

New Ground Truth wizard step in run eval flow (appears when exactly 1 session is selected)
Evaluator creation screen now shows reference input placeholders ({assertions}, {expected_tool_trajectory}, {actual_tool_trajectory}, {expected_response})
Updated default evaluator instructions to include GT placeholders

Implementation

evaluationReferenceInputs passed through to the evaluate() API call
Results include referenceInputs summary in both CLI and TUI output
Note: The second commit (temp: use middleware...) is a temporary workaround — the SDK does not yet include evaluationReferenceInputs in its model, so we inject it via Smithy middleware. Remove when SDK is updated.

Test plan

npx tsc --noEmit — clean
npm run lint — 0 errors
5 new unit tests for GT reference inputs (all pass)
Existing 33 run-eval tests continue to pass

github-actions · 2026-03-30T19:49:02Z

Package Tarball

aws-agentcore-0.4.0.tgz

How to install

npm install https://github.com/aws/agentcore-cli/releases/download/pr-732-tarball/aws-agentcore-0.4.0.tgz

github-actions · 2026-03-30T19:54:57Z

Coverage Report

Status	Category	Percentage	Covered / Total
🔵	Lines	45.98%	6575 / 14297
🔵	Statements	45.56%	6986 / 15333
🔵	Functions	44.68%	1177 / 2634
🔵	Branches	46.18%	4362 / 9444

Generated in workflow #1538 for commit d5e84ae by the Vitest Coverage Report Action

Bump @aws-sdk/client-bedrock-agentcore and @aws-sdk/client-bedrock-agentcore-control from ^3.893.0 to ^3.1020.0. The SDK now includes EvaluationReferenceInput in its model, so we pass it directly to EvaluateCommand instead of injecting via middleware.

jariy17 · 2026-03-30T20:42:49Z

Testing Results

Verified all new GT features against a deployed runtime (TempAgent in EvalCheck project):

CLI flags tested

-A, --assertion — single and multiple assertions with Builtin.GoalSuccessRate
--expected-trajectory — comma-separated tool names with Builtin.TrajectoryExactOrderMatch
--expected-response — with and without explicit -t trace ID, using Builtin.Correctness
All three combined in a single run across multiple evaluators
Baseline run (no GT inputs) — confirms referenceInputs is absent from output

Custom evaluators tested

MyEvaluatorTraceGT (TRACE level) — {expected_response} placeholder populated correctly
CustomEvaluatorSessionGT (SESSION level) — {assertions}, {expected_tool_trajectory}, {actual_tool_trajectory} placeholders populated correctly

SDK upgrade verified

@aws-sdk/client-bedrock-agentcore@^3.1020.0 — evaluationReferenceInputs passed natively via EvaluateCommand (no middleware workaround needed)

Other checks

npx tsc --noEmit — clean
npm run lint — 0 errors
All 38 unit tests pass (including 5 new GT tests)
TUI run eval wizard shows Ground Truth step when exactly 1 session is selected

Hweinstock

Some nits and question on service behavior. Otherwise lgtm.

Hweinstock

Slightly confused on the customer experience, but thats likely because I'm new to evals. The changes themselves lgtm.

Cover 13 PRs merged to main since the initial docs audit (March 28): - import subcommands: runtime, memory, evaluator, online-eval (#763, #780) - invoke/dev exec mode with --exec and --timeout (#750) - code-based evaluator support with --type, --lambda-arn, --timeout (#739) - ground truth eval inputs: --assertion, --expected-trajectory, --expected-response (#732) - memory record streaming: --data-stream-arn, --stream-content-level (#531, #534) - create --skip-install flag (#782) - fetch access --identity-name option (#774)

jariy17 · 2026-03-30T19:48:17Z

Summary

Adds ground truth reference inputs to on-demand evaluation (run eval), allowing users to specify expected agent behavior for more precise evaluation.

New CLI flags

-A, --assertion <text...> — Assertion the agent should satisfy (repeatable)
--expected-trajectory <names> — Expected tool calls in order (comma-separated)
--expected-response <text> — Expected agent response text

TUI changes

New Ground Truth wizard step in run eval flow (appears when exactly 1 session is selected)
Evaluator creation screen now shows reference input placeholders ({assertions}, {expected_tool_trajectory}, {actual_tool_trajectory}, {expected_response})
Updated default evaluator instructions to include GT placeholders

Implementation

evaluationReferenceInputs passed through to the evaluate() API call
Results include referenceInputs summary in both CLI and TUI output
Note: The second commit (temp: use middleware...) is a temporary workaround — the SDK does not yet include evaluationReferenceInputs in its model, so we inject it via Smithy middleware. Remove when SDK is updated.

Test plan

npx tsc --noEmit — clean
npm run lint — 0 errors
5 new unit tests for GT reference inputs (all pass)
Existing 33 run-eval tests continue to pass

github-actions · 2026-03-30T19:49:02Z

Package Tarball

aws-agentcore-0.4.0.tgz

How to install

npm install https://github.com/aws/agentcore-cli/releases/download/pr-732-tarball/aws-agentcore-0.4.0.tgz

github-actions · 2026-03-30T19:54:57Z

Coverage Report

Status	Category	Percentage	Covered / Total
🔵	Lines	45.98%	6575 / 14297
🔵	Statements	45.56%	6986 / 15333
🔵	Functions	44.68%	1177 / 2634
🔵	Branches	46.18%	4362 / 9444

Generated in workflow #1538 for commit d5e84ae by the Vitest Coverage Report Action

Bump @aws-sdk/client-bedrock-agentcore and @aws-sdk/client-bedrock-agentcore-control from ^3.893.0 to ^3.1020.0. The SDK now includes EvaluationReferenceInput in its model, so we pass it directly to EvaluateCommand instead of injecting via middleware.

jariy17 · 2026-03-30T20:42:49Z

Testing Results

Verified all new GT features against a deployed runtime (TempAgent in EvalCheck project):

CLI flags tested

-A, --assertion — single and multiple assertions with Builtin.GoalSuccessRate
--expected-trajectory — comma-separated tool names with Builtin.TrajectoryExactOrderMatch
--expected-response — with and without explicit -t trace ID, using Builtin.Correctness
All three combined in a single run across multiple evaluators
Baseline run (no GT inputs) — confirms referenceInputs is absent from output

Custom evaluators tested

MyEvaluatorTraceGT (TRACE level) — {expected_response} placeholder populated correctly
CustomEvaluatorSessionGT (SESSION level) — {assertions}, {expected_tool_trajectory}, {actual_tool_trajectory} placeholders populated correctly

SDK upgrade verified

@aws-sdk/client-bedrock-agentcore@^3.1020.0 — evaluationReferenceInputs passed natively via EvaluateCommand (no middleware workaround needed)

Other checks

npx tsc --noEmit — clean
npm run lint — 0 errors
All 38 unit tests pass (including 5 new GT tests)
TUI run eval wizard shows Ground Truth step when exactly 1 session is selected

Hweinstock

Some nits and question on service behavior. Otherwise lgtm.

Hweinstock

Slightly confused on the customer experience, but thats likely because I'm new to evals. The changes themselves lgtm.

Cover 13 PRs merged to main since the initial docs audit (March 28): - import subcommands: runtime, memory, evaluator, online-eval (#763, #780) - invoke/dev exec mode with --exec and --timeout (#750) - code-based evaluator support with --type, --lambda-arn, --timeout (#739) - ground truth eval inputs: --assertion, --expected-trajectory, --expected-response (#732) - memory record streaming: --data-stream-arn, --stream-content-level (#531, #534) - create --skip-install flag (#782) - fetch access --identity-name option (#774)

jariy17 · 2026-03-30T19:48:17Z

Summary

Adds ground truth reference inputs to on-demand evaluation (run eval), allowing users to specify expected agent behavior for more precise evaluation.

New CLI flags

-A, --assertion <text...> — Assertion the agent should satisfy (repeatable)
--expected-trajectory <names> — Expected tool calls in order (comma-separated)
--expected-response <text> — Expected agent response text

TUI changes

New Ground Truth wizard step in run eval flow (appears when exactly 1 session is selected)
Evaluator creation screen now shows reference input placeholders ({assertions}, {expected_tool_trajectory}, {actual_tool_trajectory}, {expected_response})
Updated default evaluator instructions to include GT placeholders

Implementation

evaluationReferenceInputs passed through to the evaluate() API call
Results include referenceInputs summary in both CLI and TUI output
Note: The second commit (temp: use middleware...) is a temporary workaround — the SDK does not yet include evaluationReferenceInputs in its model, so we inject it via Smithy middleware. Remove when SDK is updated.

Test plan

npx tsc --noEmit — clean
npm run lint — 0 errors
5 new unit tests for GT reference inputs (all pass)
Existing 33 run-eval tests continue to pass

github-actions · 2026-03-30T19:49:02Z

Package Tarball

aws-agentcore-0.4.0.tgz

How to install

npm install https://github.com/aws/agentcore-cli/releases/download/pr-732-tarball/aws-agentcore-0.4.0.tgz

github-actions · 2026-03-30T19:54:57Z

Coverage Report

Status	Category	Percentage	Covered / Total
🔵	Lines	45.98%	6575 / 14297
🔵	Statements	45.56%	6986 / 15333
🔵	Functions	44.68%	1177 / 2634
🔵	Branches	46.18%	4362 / 9444

Generated in workflow #1538 for commit d5e84ae by the Vitest Coverage Report Action

Bump @aws-sdk/client-bedrock-agentcore and @aws-sdk/client-bedrock-agentcore-control from ^3.893.0 to ^3.1020.0. The SDK now includes EvaluationReferenceInput in its model, so we pass it directly to EvaluateCommand instead of injecting via middleware.

jariy17 · 2026-03-30T20:42:49Z

Testing Results

Verified all new GT features against a deployed runtime (TempAgent in EvalCheck project):

CLI flags tested

-A, --assertion — single and multiple assertions with Builtin.GoalSuccessRate
--expected-trajectory — comma-separated tool names with Builtin.TrajectoryExactOrderMatch
--expected-response — with and without explicit -t trace ID, using Builtin.Correctness
All three combined in a single run across multiple evaluators
Baseline run (no GT inputs) — confirms referenceInputs is absent from output

Custom evaluators tested

MyEvaluatorTraceGT (TRACE level) — {expected_response} placeholder populated correctly
CustomEvaluatorSessionGT (SESSION level) — {assertions}, {expected_tool_trajectory}, {actual_tool_trajectory} placeholders populated correctly

SDK upgrade verified

@aws-sdk/client-bedrock-agentcore@^3.1020.0 — evaluationReferenceInputs passed natively via EvaluateCommand (no middleware workaround needed)

Other checks

npx tsc --noEmit — clean
npm run lint — 0 errors
All 38 unit tests pass (including 5 new GT tests)
TUI run eval wizard shows Ground Truth step when exactly 1 session is selected

Hweinstock

Some nits and question on service behavior. Otherwise lgtm.

Hweinstock

Slightly confused on the customer experience, but thats likely because I'm new to evals. The changes themselves lgtm.

Cover 13 PRs merged to main since the initial docs audit (March 28): - import subcommands: runtime, memory, evaluator, online-eval (#763, #780) - invoke/dev exec mode with --exec and --timeout (#750) - code-based evaluator support with --type, --lambda-arn, --timeout (#739) - ground truth eval inputs: --assertion, --expected-trajectory, --expected-response (#732) - memory record streaming: --data-stream-arn, --stream-content-level (#531, #534) - create --skip-install flag (#782) - fetch access --identity-name option (#774)

jariy17 · 2026-03-30T19:48:17Z

Summary

Adds ground truth reference inputs to on-demand evaluation (run eval), allowing users to specify expected agent behavior for more precise evaluation.

New CLI flags

-A, --assertion <text...> — Assertion the agent should satisfy (repeatable)
--expected-trajectory <names> — Expected tool calls in order (comma-separated)
--expected-response <text> — Expected agent response text

TUI changes

New Ground Truth wizard step in run eval flow (appears when exactly 1 session is selected)
Evaluator creation screen now shows reference input placeholders ({assertions}, {expected_tool_trajectory}, {actual_tool_trajectory}, {expected_response})
Updated default evaluator instructions to include GT placeholders

Implementation

evaluationReferenceInputs passed through to the evaluate() API call
Results include referenceInputs summary in both CLI and TUI output
Note: The second commit (temp: use middleware...) is a temporary workaround — the SDK does not yet include evaluationReferenceInputs in its model, so we inject it via Smithy middleware. Remove when SDK is updated.

Test plan

npx tsc --noEmit — clean
npm run lint — 0 errors
5 new unit tests for GT reference inputs (all pass)
Existing 33 run-eval tests continue to pass

github-actions · 2026-03-30T19:49:02Z

Package Tarball

aws-agentcore-0.4.0.tgz

How to install

npm install https://github.com/aws/agentcore-cli/releases/download/pr-732-tarball/aws-agentcore-0.4.0.tgz

github-actions · 2026-03-30T19:54:57Z

Coverage Report

Status	Category	Percentage	Covered / Total
🔵	Lines	45.98%	6575 / 14297
🔵	Statements	45.56%	6986 / 15333
🔵	Functions	44.68%	1177 / 2634
🔵	Branches	46.18%	4362 / 9444

Generated in workflow #1538 for commit d5e84ae by the Vitest Coverage Report Action

Bump @aws-sdk/client-bedrock-agentcore and @aws-sdk/client-bedrock-agentcore-control from ^3.893.0 to ^3.1020.0. The SDK now includes EvaluationReferenceInput in its model, so we pass it directly to EvaluateCommand instead of injecting via middleware.

jariy17 · 2026-03-30T20:42:49Z

Testing Results

Verified all new GT features against a deployed runtime (TempAgent in EvalCheck project):

CLI flags tested

-A, --assertion — single and multiple assertions with Builtin.GoalSuccessRate
--expected-trajectory — comma-separated tool names with Builtin.TrajectoryExactOrderMatch
--expected-response — with and without explicit -t trace ID, using Builtin.Correctness
All three combined in a single run across multiple evaluators
Baseline run (no GT inputs) — confirms referenceInputs is absent from output

Custom evaluators tested

MyEvaluatorTraceGT (TRACE level) — {expected_response} placeholder populated correctly
CustomEvaluatorSessionGT (SESSION level) — {assertions}, {expected_tool_trajectory}, {actual_tool_trajectory} placeholders populated correctly

SDK upgrade verified

@aws-sdk/client-bedrock-agentcore@^3.1020.0 — evaluationReferenceInputs passed natively via EvaluateCommand (no middleware workaround needed)

Other checks

npx tsc --noEmit — clean
npm run lint — 0 errors
All 38 unit tests pass (including 5 new GT tests)
TUI run eval wizard shows Ground Truth step when exactly 1 session is selected

Hweinstock

Some nits and question on service behavior. Otherwise lgtm.

Hweinstock

Slightly confused on the customer experience, but thats likely because I'm new to evals. The changes themselves lgtm.

Cover 13 PRs merged to main since the initial docs audit (March 28): - import subcommands: runtime, memory, evaluator, online-eval (#763, #780) - invoke/dev exec mode with --exec and --timeout (#750) - code-based evaluator support with --type, --lambda-arn, --timeout (#739) - ground truth eval inputs: --assertion, --expected-trajectory, --expected-response (#732) - memory record streaming: --data-stream-arn, --stream-content-level (#531, #534) - create --skip-install flag (#782) - fetch access --identity-name option (#774)

jariy17 · 2026-03-30T19:48:17Z

Summary

Adds ground truth reference inputs to on-demand evaluation (run eval), allowing users to specify expected agent behavior for more precise evaluation.

New CLI flags

-A, --assertion <text...> — Assertion the agent should satisfy (repeatable)
--expected-trajectory <names> — Expected tool calls in order (comma-separated)
--expected-response <text> — Expected agent response text

TUI changes

New Ground Truth wizard step in run eval flow (appears when exactly 1 session is selected)
Evaluator creation screen now shows reference input placeholders ({assertions}, {expected_tool_trajectory}, {actual_tool_trajectory}, {expected_response})
Updated default evaluator instructions to include GT placeholders

Implementation

evaluationReferenceInputs passed through to the evaluate() API call
Results include referenceInputs summary in both CLI and TUI output
Note: The second commit (temp: use middleware...) is a temporary workaround — the SDK does not yet include evaluationReferenceInputs in its model, so we inject it via Smithy middleware. Remove when SDK is updated.

Test plan

npx tsc --noEmit — clean
npm run lint — 0 errors
5 new unit tests for GT reference inputs (all pass)
Existing 33 run-eval tests continue to pass

github-actions · 2026-03-30T19:49:02Z

Package Tarball

aws-agentcore-0.4.0.tgz

How to install

npm install https://github.com/aws/agentcore-cli/releases/download/pr-732-tarball/aws-agentcore-0.4.0.tgz

github-actions · 2026-03-30T19:54:57Z

Coverage Report

Status	Category	Percentage	Covered / Total
🔵	Lines	45.98%	6575 / 14297
🔵	Statements	45.56%	6986 / 15333
🔵	Functions	44.68%	1177 / 2634
🔵	Branches	46.18%	4362 / 9444

Generated in workflow #1538 for commit d5e84ae by the Vitest Coverage Report Action

Bump @aws-sdk/client-bedrock-agentcore and @aws-sdk/client-bedrock-agentcore-control from ^3.893.0 to ^3.1020.0. The SDK now includes EvaluationReferenceInput in its model, so we pass it directly to EvaluateCommand instead of injecting via middleware.

jariy17 · 2026-03-30T20:42:49Z

Testing Results

Verified all new GT features against a deployed runtime (TempAgent in EvalCheck project):

CLI flags tested

-A, --assertion — single and multiple assertions with Builtin.GoalSuccessRate
--expected-trajectory — comma-separated tool names with Builtin.TrajectoryExactOrderMatch
--expected-response — with and without explicit -t trace ID, using Builtin.Correctness
All three combined in a single run across multiple evaluators
Baseline run (no GT inputs) — confirms referenceInputs is absent from output

Custom evaluators tested

MyEvaluatorTraceGT (TRACE level) — {expected_response} placeholder populated correctly
CustomEvaluatorSessionGT (SESSION level) — {assertions}, {expected_tool_trajectory}, {actual_tool_trajectory} placeholders populated correctly

SDK upgrade verified

@aws-sdk/client-bedrock-agentcore@^3.1020.0 — evaluationReferenceInputs passed natively via EvaluateCommand (no middleware workaround needed)

Other checks

npx tsc --noEmit — clean
npm run lint — 0 errors
All 38 unit tests pass (including 5 new GT tests)
TUI run eval wizard shows Ground Truth step when exactly 1 session is selected

Hweinstock

Some nits and question on service behavior. Otherwise lgtm.

Hweinstock

Slightly confused on the customer experience, but thats likely because I'm new to evals. The changes themselves lgtm.

Cover 13 PRs merged to main since the initial docs audit (March 28): - import subcommands: runtime, memory, evaluator, online-eval (#763, #780) - invoke/dev exec mode with --exec and --timeout (#750) - code-based evaluator support with --type, --lambda-arn, --timeout (#739) - ground truth eval inputs: --assertion, --expected-trajectory, --expected-response (#732) - memory record streaming: --data-stream-arn, --stream-content-level (#531, #534) - create --skip-install flag (#782) - fetch access --identity-name option (#774)

jariy17 · 2026-03-30T19:48:17Z

Summary

Adds ground truth reference inputs to on-demand evaluation (run eval), allowing users to specify expected agent behavior for more precise evaluation.

New CLI flags

-A, --assertion <text...> — Assertion the agent should satisfy (repeatable)
--expected-trajectory <names> — Expected tool calls in order (comma-separated)
--expected-response <text> — Expected agent response text

TUI changes

New Ground Truth wizard step in run eval flow (appears when exactly 1 session is selected)
Evaluator creation screen now shows reference input placeholders ({assertions}, {expected_tool_trajectory}, {actual_tool_trajectory}, {expected_response})
Updated default evaluator instructions to include GT placeholders

Implementation

evaluationReferenceInputs passed through to the evaluate() API call
Results include referenceInputs summary in both CLI and TUI output
Note: The second commit (temp: use middleware...) is a temporary workaround — the SDK does not yet include evaluationReferenceInputs in its model, so we inject it via Smithy middleware. Remove when SDK is updated.

Test plan

npx tsc --noEmit — clean
npm run lint — 0 errors
5 new unit tests for GT reference inputs (all pass)
Existing 33 run-eval tests continue to pass

github-actions · 2026-03-30T19:49:02Z

Package Tarball

aws-agentcore-0.4.0.tgz

How to install

npm install https://github.com/aws/agentcore-cli/releases/download/pr-732-tarball/aws-agentcore-0.4.0.tgz

github-actions · 2026-03-30T19:54:57Z

Coverage Report

Status	Category	Percentage	Covered / Total
🔵	Lines	45.98%	6575 / 14297
🔵	Statements	45.56%	6986 / 15333
🔵	Functions	44.68%	1177 / 2634
🔵	Branches	46.18%	4362 / 9444

Generated in workflow #1538 for commit d5e84ae by the Vitest Coverage Report Action

Bump @aws-sdk/client-bedrock-agentcore and @aws-sdk/client-bedrock-agentcore-control from ^3.893.0 to ^3.1020.0. The SDK now includes EvaluationReferenceInput in its model, so we pass it directly to EvaluateCommand instead of injecting via middleware.

jariy17 · 2026-03-30T20:42:49Z

Testing Results

Verified all new GT features against a deployed runtime (TempAgent in EvalCheck project):

CLI flags tested

-A, --assertion — single and multiple assertions with Builtin.GoalSuccessRate
--expected-trajectory — comma-separated tool names with Builtin.TrajectoryExactOrderMatch
--expected-response — with and without explicit -t trace ID, using Builtin.Correctness
All three combined in a single run across multiple evaluators
Baseline run (no GT inputs) — confirms referenceInputs is absent from output

Custom evaluators tested

MyEvaluatorTraceGT (TRACE level) — {expected_response} placeholder populated correctly
CustomEvaluatorSessionGT (SESSION level) — {assertions}, {expected_tool_trajectory}, {actual_tool_trajectory} placeholders populated correctly

SDK upgrade verified

@aws-sdk/client-bedrock-agentcore@^3.1020.0 — evaluationReferenceInputs passed natively via EvaluateCommand (no middleware workaround needed)

Other checks

npx tsc --noEmit — clean
npm run lint — 0 errors
All 38 unit tests pass (including 5 new GT tests)
TUI run eval wizard shows Ground Truth step when exactly 1 session is selected

Hweinstock

Some nits and question on service behavior. Otherwise lgtm.

Hweinstock

Slightly confused on the customer experience, but thats likely because I'm new to evals. The changes themselves lgtm.

Cover 13 PRs merged to main since the initial docs audit (March 28): - import subcommands: runtime, memory, evaluator, online-eval (#763, #780) - invoke/dev exec mode with --exec and --timeout (#750) - code-based evaluator support with --type, --lambda-arn, --timeout (#739) - ground truth eval inputs: --assertion, --expected-trajectory, --expected-response (#732) - memory record streaming: --data-stream-arn, --stream-content-level (#531, #534) - create --skip-install flag (#782) - fetch access --identity-name option (#774)

feat: add ground truth reference inputs for on-demand evaluation
0252612

jariy17 requested a review from a teamMarch 30, 2026 19:48

github-actionsBot added the size/l PR size: L label Mar 30, 2026

jariy17 temporarily deployed to e2e-testing March 30, 2026 19:48 — with GitHub Actions Inactive

jariy17 force-pushed the feat/GT_Support branch from 1000390 to ebbe38aCompare March 30, 2026 20:11

github-actionsBot added size/l PR size: L and removed size/l PR size: L labels Mar 30, 2026

jariy17 temporarily deployed to e2e-testing March 30, 2026 20:11 — with GitHub Actions Inactive

jariy17 commented Mar 30, 2026
View reviewed changes

Comment threadsrc/cli/aws/agentcore.ts Outdated

awsswarnim approved these changes Mar 30, 2026
View reviewed changes

Hweinstock reviewed Mar 30, 2026
View reviewed changes

Comment threadsrc/cli/commands/run/command.tsx
Comment threadsrc/cli/commands/run/command.tsx
Comment threadsrc/cli/operations/eval/run-eval.ts
Comment threadsrc/cli/operations/eval/run-eval.ts
Comment threadsrc/cli/operations/eval/run-eval.ts
Comment threadsrc/cli/tui/screens/evaluator/types.ts

Hweinstock previously approved these changes Mar 30, 2026
View reviewed changes

fix: address PR review — combine import, add single-session GT guard
d5e84ae

jariy17 dismissed Hweinstock’s stale review via d5e84aeMarch 30, 2026 22:48

github-actionsBot removed the size/l PR size: L label Mar 30, 2026

jariy17 temporarily deployed to e2e-testing March 30, 2026 22:48 — with GitHub Actions Inactive

github-actionsBot added the size/l PR size: L label Mar 30, 2026

Hweinstock approved these changes Mar 30, 2026
View reviewed changes

aws deleted a comment from Hweinstock Mar 30, 2026

jariy17 merged commit 01623ff into mainMar 30, 2026
22 checks passed

jariy17 deleted the feat/GT_Support branch March 30, 2026 23:03

aidandaly24 mentioned this pull request Apr 8, 2026
docs: fix 30 documentation inaccuracies found by source code audit #697
Merged
15 tasks