feat(eval): batch-evaluation simulate — each example owns its invoker by jariy17 · Pull Request #2032 · aws/agentcore-cli

jariy17 · 2026-08-18T22:40:03Z

What

Adds batch-evaluation simulate to replay a dataset against a runtime and grade the resulting sessions.

Each dataset example is a self-running object that owns its own invoker (RunContext). invokeDataset builds one invoker per session and passes it into the example's run(). The example performs the invocation; the machine supplies the invoker and collects the result.

The invoker seam

A dataset example knows what to send and how to turn the responses into ground truth, but not how to reach the runtime. Its RunContext owns that runtime-specific invocation:

// src/core/eval/invokeDataset/example/types.tsexporttypeTurnResult={text: string};exporttypeRunContext={invokeOnce(input: string): Promise<TurnResult>};exportinterfaceExample{readonlyexampleId: string;readonlyschemaType: DatasetSchemaType;// Replay this example against the runtime `ctx` reaches, return neutral ground truth.run(ctx: RunContext): Promise<InlineGroundTruth|undefined>;}

invokeDataset resolves the runtime once, then builds a distinct invoker for each client-generated session and passes it to that example:

constresults=awaitrunExamples(examples,async(example)=>{constctx: RunContext={invokeOnce: async(input)=>{constres=awaitinvokeRuntime(deps,{/* resolved runtime, session, and rendered payload */},options,signal,);return{text: /* drained response body */};},};returnexample.run(ctx);});

The example decides how many times to call invokeOnce and in what order. Its invoker decides how each call reaches the runtime for that session.

Layout

src/core/eval/invokeDataset/
├── load.ts DatasetLoader — pure JSONL parse → shape-classify → new
├── run.ts runExamples — bounded-concurrency pool, failure isolation
├── invokeDataset.test.ts end-to-end golden coverage of the whole path
└── example/
├── types.ts Example interface, RunContext (the invoker), TurnResult
├── predefined.ts PredefinedExample — replays scripted turns, builds ground truth
└── simulated.ts SimulatedExample — not shipped; throws at construction

Plus src/core/invokeRuntime.ts, which extracts runtime invocation from runtime.tsx for reuse by both RuntimeClient and invokeDataset, and src/handlers/eval/batch-evaluation/simulate/index.tsx, which composes invokeDataset with startBatchEvaluation.

Testing

bun run typecheck passes.
bun test: 1544 pass, 0 fail.
invokeDataset.test.ts drives the real EvalClient.invokeDataset over a fake AWS layer with golden fixtures. Its snapshot covers created sessions and inline ground truth across every supported variation while exercising the loader, example classes, concurrency pool, template rendering, and runtime invocation.
simulate.test.tsx snapshots the handler's wrapped sessionMetadata.

codecov-commenter · 2026-08-18T22:42:42Z

Codecov Report

❌ Patch coverage is 96.58444% with 18 lines in your changes missing coverage. Please review.
✅ Project coverage is 97.33%. Comparing base (768ef10) to head (27c810f).

Files with missing lines	Patch %	Lines
src/core/eval.tsx	85.71%	14 Missing ⚠️
src/core/eval/invokeDataset/template.ts	91.30%	2 Missing ⚠️
src/core/eval/invokeDataset/example/simulated.ts	92.30%	1 Missing ⚠️
src/core/invokeRuntime.ts	99.35%	1 Missing ⚠️

Additional details and impacted files

@@ Coverage Diff @@## refactor #2032 +/- ##
============================================
- Coverage 97.36% 97.33% -0.04% 
============================================
Files 410 417 +7 Lines 24796 25181 +385 ============================================
+ Hits 24142 24509 +367 - Misses 654 672 +18

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Hweinstock

didn't get to all the examples work yet, but had a few small comments and a question on how we can simplify testing, because it feels pretty gnarly rn.

Hweinstock · 2026-08-19T19:55:13Z

@@ -0,0 +1,33 @@
+import { InputValidationError } from "../errors";


should we keep this inside evals until there is an opportunity to re-use it? I feel like the io directory should be reserved for shared abstractions.

Yeah, I was thinking the invoke handler could use this but lets leave it in the simulate handler for now.

Hweinstock · 2026-08-19T19:55:23Z

+ try {
+ parsed = JSON.parse(template);
+ } catch {
+ throw new InputValidationError(`--${flagName} must be valid JSON`);


should we wire the cause here?

Hweinstock · 2026-08-19T20:02:10Z

+ qualifier?: string;
+ payloadTemplate: string; // e.g. {"prompt":"{input}"} — {input} is the example's turn input
+ headers?: [string, string][];
+ bearerToken?: string;


q: is there a reason bearerToken is treated different from other headers?

It's the discriminator that selects the auth path, not an application header. Its presence routes the invoke to CUSTOM_JWT path (src/core/invokeRuntime.ts:73)

Hweinstock · 2026-08-19T20:07:33Z

+ flag("dataset-version", "dataset version (with a dataset id)", z.string().optional()),
+ flag("evaluator", "evaluator id(s) to apply", z.array(z.string()).optional()),
+ flag("name", "batch evaluation name (unique in the account)", z.string().optional()),
+ flag("description", "optional description", z.string().optional()),


what exactly is this describing? the simulation itself?

This should be batch-evaluation description. I'll rename this batch-eval-desc, same with name too.

Hweinstock · 2026-08-19T20:08:58Z

+ throw new InputValidationError("required option '--name <name>' not specified");
+
+ // Ctrl-C aborts the run (invokes, the ingestion wait, the dataset download).
+ const controller = new AbortController();


ahhh, theres a shared abstraction for this, but looks like it hasn't been merged yet :(
#1986
maybe we can swap it over as a follow-up once its merged.

looks like it was merged, so we should be able to leverage that shared abstraction now.
#1986

Hweinstock · 2026-08-19T20:33:35Z

+ const groundTruth = await example.run(ctx);
+ return { exampleId: example.exampleId, sessionId, groundTruth };
+ } catch (error) {
+ this.logger.debug(


instead of logging and rethrowing, is there a way to enrich the error thrown to avoid noise?

Hweinstock · 2026-08-19T20:34:33Z

+
+ // AgentCore emits spans ~30s-3min after invoke; grade too early and it reads an empty
+ // log group and fails every session. Disabled via SIMULATE_INGESTION_WAIT_MS=0 (tests).
+ const waitMs = Number(process.env.SIMULATE_INGESTION_WAIT_MS ?? 180_000);


is there anything we can poll on instead of a static wait time?

I didn't poll right away because some traces might still be arriving, and I didn't want us to end up with incomplete session data.

how do we know that there aren't traces still coming in after 3 min mark?

Hweinstock · 2026-08-19T20:44:21Z

+
+// A fake AWS layer: control resolves the runtime, data answers each invoke. Records every
+// payload it was asked to send, and per `opts` can fail or delay specific invokes.
+function fakeClients(opts: { fail?: (payload: string) => boolean; delayMs?: number } = {}): {


this feels like a really complex testing setup. Is there a simpler way? I'm wondering if there's a structural change we could make to simplify here.

Can I fixed these tests as a follow up?

Hweinstock · 2026-08-19T20:44:48Z

+ let parsed: unknown;
+ try {
+ parsed = JSON.parse(trimmed);
+ } catch {


should we wire the cause here?

Hweinstock · 2026-08-19T20:47:11Z

+import type { DatasetSchemaType } from "@aws-sdk/client-bedrock-agentcore-control";
+import type { InlineGroundTruth } from "@aws-sdk/client-bedrock-agentcore";
+
+// A record, not a bare string, so a future tool-branching type can widen it by a field.


i feel like the code explains this comment.

Hweinstock

only blocking question is why we do an explicit wait, rest could be polished later/in a follow-up. Is there a way we can be sure the spans we need exist before evaluating or is there a reason we don't have to be?

Hweinstock · 2026-08-24T14:39:15Z

+ throw new InputValidationError("required option '--name <name>' not specified");
+
+ // Ctrl-C aborts the run (invokes, the ingestion wait, the dataset download).
+ const controller = new AbortController();


looks like it was merged, so we should be able to leverage that shared abstraction now.
#1986

Hweinstock · 2026-08-24T14:42:18Z

+import { coreOptsFromCtx } from "../../../utils";
+import { parseRuntimeInvokeHeaders } from "../../../runtime/invoke/request";
+
+// Composes invokeDataset (replay) → startBatchEvaluation (grade). Invoke flags mirror


is the comment describing the implementation of the handler? What value does it provide over the code?

Hweinstock · 2026-08-24T16:13:18Z

+ // Enrich with the example identity so the dropped-invoke reason is self-describing
+ // in firstError, instead of a bare transport message logged separately.
+ const cause = error instanceof Error ? error : new Error(String(error));
+ throw new Error(


to confirm my understanding, this error never hits telemetry because an individual example failing does not bubble up so this error is purely for the user message it creates?

Correct. An individual example failing is caught and dropped by runExamples rather than rethrown, so it never reaches telemetry — it only feeds the user-facing message on a total failure. That was intentional: a run can be hundreds of examples, and I didn't want per-example failures flooding our telemetry.

Hweinstock · 2026-08-24T16:13:49Z

+
+ // AgentCore emits spans ~30s-3min after invoke; grade too early and it reads an empty
+ // log group and fails every session. Disabled via SIMULATE_INGESTION_WAIT_MS=0 (tests).
+ const waitMs = Number(process.env.SIMULATE_INGESTION_WAIT_MS ?? 180_000);


how do we know that there aren't traces still coming in after 3 min mark?

Hweinstock · 2026-08-24T16:17:43Z

+ try {
+ headers.set("Authorization", `Bearer ${bearerToken}`);
+ } catch {
+ throw new InputValidationError("Invalid bearer token");


should we wire the cause up to these errors to get the stack trace? (same with 132, 155 below)

Hweinstock · 2026-08-24T16:18:43Z

+ logger
+ .child({
+ errorName:
+ error instanceof TypeError


is there a reason error instanceof error ? error.name : 'error' doesn't work here?

I'll look into this in the next PR.

Hweinstock · 2026-08-24T16:21:33Z

+};
+
+// InvokeDatasetResult reports the created sessions plus how many examples were invoked
+// vs dropped (a failed invoke is skipped, not fatal). firstError explains a total failure.


does firstError explain a total failure? I thought it stored the first failed example's error?

WE will remove it later. Yes lets do a follow up on this.

Hweinstock · 2026-08-24T16:27:49Z

@@ -0,0 +1,263 @@
+// Disables the post-invoke span-ingestion wait so the replay returns immediately.
+process.env.SIMULATE_INGESTION_WAIT_MS = "0";


would this be a good application of global config? this might help this behavior be more discoverable by other tests that might need the same.

So add to the bunfig.toml to preload a script that run this:
./global-mocks.ts
// Global mocksimport{mock}from"bun:test";// Mock environment variablesprocess.env.SIMULATE_INGESTION_WAIT_MS="0";
bunfig.toml
[test]preload=["./global-mocks.ts"]
ref
Yes I could do that in the follow up Pr.

we should be able to inject the global config down through core with the data we want.

ah that make sense

Hweinstock · 2026-08-24T16:32:38Z

+ return this.groundTruth();
+ }
+
+ // Emit every turn (carrying its prompt), not just those with an expectation: filtering


i'm not quite sure what this comment is saying?

Ill remove it.

Hweinstock · 2026-08-24T21:27:20Z

Is there a way we can be sure the spans we need exist before evaluating or is there a reason we don't have to be?

from chatting offline, sounds like the wait approach is shared by the eval time. Still unsure how they can be confident the evaluator results are not incomplete, but approving since its an existing solution.

…example classes

- move renderJsonTemplate out of shared src/io into core/eval/invokeDataset - invokeRuntime: raw TypeError/Error -> InputValidationError/RuntimeInvokeResponseError - wire error causes in template + dataset JSON parse - invokeDataset: enrich per-example invoke failure instead of log+rethrow - simulate: bubble Ctrl-C cancellation (telemetry) instead of quiet return; clarify --description help; TODO(#1986) shared abort helper - drop type-guaranteed 'no leak' test; keep AbortSignal wiring in composition test - trim stale/redundant comments (runtime.tsx, invokeRuntime DTO note, TurnResult)

…in payload-template Bug bash (real exploratory-account run) surfaced two defects in the shared invokeDataset path: 1. PredefinedExample blind-cast assertions/expected_trajectory as string[] with no load-time validation (unlike turns). A non-array value passed load, burned a live paid invoke, then threw a raw '.map is not a function' mislabeled 'failed to invoke'. Now validated in the constructor -> clean InputValidationError before any invoke. 2. A --payload-template with no {input} placeholder was silently accepted, wasting a full ~3-min replay on a constant payload. Now rejected up front in invokeDataset.

nborges-aws · 2026-08-24T22:40:49Z

+ const sessionId = randomUUID();
+ const ctx: RunContext = {
+ invokeOnce: async (payload) => {
+ const response = await invokeRuntime(


Comparing this to src/handlers/runtime/invoke/request.ts, and I notice that the code calls normalizeRuntimeInvokeRequest to do some validation before invoking. Do you think we need something similar here?

nborges-aws

LGTM. One comment with a question for you

jariy17 · 2026-08-18T22:40:03Z

What

Adds batch-evaluation simulate to replay a dataset against a runtime and grade the resulting sessions.

Each dataset example is a self-running object that owns its own invoker (RunContext). invokeDataset builds one invoker per session and passes it into the example's run(). The example performs the invocation; the machine supplies the invoker and collects the result.

The invoker seam

A dataset example knows what to send and how to turn the responses into ground truth, but not how to reach the runtime. Its RunContext owns that runtime-specific invocation:

// src/core/eval/invokeDataset/example/types.tsexporttypeTurnResult={text: string};exporttypeRunContext={invokeOnce(input: string): Promise<TurnResult>};exportinterfaceExample{readonlyexampleId: string;readonlyschemaType: DatasetSchemaType;// Replay this example against the runtime `ctx` reaches, return neutral ground truth.run(ctx: RunContext): Promise<InlineGroundTruth|undefined>;}

invokeDataset resolves the runtime once, then builds a distinct invoker for each client-generated session and passes it to that example:

constresults=awaitrunExamples(examples,async(example)=>{constctx: RunContext={invokeOnce: async(input)=>{constres=awaitinvokeRuntime(deps,{/* resolved runtime, session, and rendered payload */},options,signal,);return{text: /* drained response body */};},};returnexample.run(ctx);});

The example decides how many times to call invokeOnce and in what order. Its invoker decides how each call reaches the runtime for that session.

Layout

src/core/eval/invokeDataset/
├── load.ts DatasetLoader — pure JSONL parse → shape-classify → new
├── run.ts runExamples — bounded-concurrency pool, failure isolation
├── invokeDataset.test.ts end-to-end golden coverage of the whole path
└── example/
├── types.ts Example interface, RunContext (the invoker), TurnResult
├── predefined.ts PredefinedExample — replays scripted turns, builds ground truth
└── simulated.ts SimulatedExample — not shipped; throws at construction

Plus src/core/invokeRuntime.ts, which extracts runtime invocation from runtime.tsx for reuse by both RuntimeClient and invokeDataset, and src/handlers/eval/batch-evaluation/simulate/index.tsx, which composes invokeDataset with startBatchEvaluation.

Testing

bun run typecheck passes.
bun test: 1544 pass, 0 fail.
invokeDataset.test.ts drives the real EvalClient.invokeDataset over a fake AWS layer with golden fixtures. Its snapshot covers created sessions and inline ground truth across every supported variation while exercising the loader, example classes, concurrency pool, template rendering, and runtime invocation.
simulate.test.tsx snapshots the handler's wrapped sessionMetadata.

codecov-commenter · 2026-08-18T22:42:42Z

Codecov Report

❌ Patch coverage is 96.58444% with 18 lines in your changes missing coverage. Please review.
✅ Project coverage is 97.33%. Comparing base (768ef10) to head (27c810f).

Files with missing lines	Patch %	Lines
src/core/eval.tsx	85.71%	14 Missing ⚠️
src/core/eval/invokeDataset/template.ts	91.30%	2 Missing ⚠️
src/core/eval/invokeDataset/example/simulated.ts	92.30%	1 Missing ⚠️
src/core/invokeRuntime.ts	99.35%	1 Missing ⚠️

Additional details and impacted files

@@ Coverage Diff @@## refactor #2032 +/- ##
============================================
- Coverage 97.36% 97.33% -0.04% 
============================================
Files 410 417 +7 Lines 24796 25181 +385 ============================================
+ Hits 24142 24509 +367 - Misses 654 672 +18

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Hweinstock

didn't get to all the examples work yet, but had a few small comments and a question on how we can simplify testing, because it feels pretty gnarly rn.

Hweinstock · 2026-08-19T19:55:13Z

@@ -0,0 +1,33 @@
+import { InputValidationError } from "../errors";


should we keep this inside evals until there is an opportunity to re-use it? I feel like the io directory should be reserved for shared abstractions.

Yeah, I was thinking the invoke handler could use this but lets leave it in the simulate handler for now.

Hweinstock · 2026-08-19T19:55:23Z

+ try {
+ parsed = JSON.parse(template);
+ } catch {
+ throw new InputValidationError(`--${flagName} must be valid JSON`);


should we wire the cause here?

Hweinstock · 2026-08-19T20:02:10Z

+ qualifier?: string;
+ payloadTemplate: string; // e.g. {"prompt":"{input}"} — {input} is the example's turn input
+ headers?: [string, string][];
+ bearerToken?: string;


q: is there a reason bearerToken is treated different from other headers?

It's the discriminator that selects the auth path, not an application header. Its presence routes the invoke to CUSTOM_JWT path (src/core/invokeRuntime.ts:73)

Hweinstock · 2026-08-19T20:07:33Z

+ flag("dataset-version", "dataset version (with a dataset id)", z.string().optional()),
+ flag("evaluator", "evaluator id(s) to apply", z.array(z.string()).optional()),
+ flag("name", "batch evaluation name (unique in the account)", z.string().optional()),
+ flag("description", "optional description", z.string().optional()),


what exactly is this describing? the simulation itself?

This should be batch-evaluation description. I'll rename this batch-eval-desc, same with name too.

Hweinstock · 2026-08-19T20:08:58Z

+ throw new InputValidationError("required option '--name <name>' not specified");
+
+ // Ctrl-C aborts the run (invokes, the ingestion wait, the dataset download).
+ const controller = new AbortController();


ahhh, theres a shared abstraction for this, but looks like it hasn't been merged yet :(
#1986
maybe we can swap it over as a follow-up once its merged.

looks like it was merged, so we should be able to leverage that shared abstraction now.
#1986

Hweinstock · 2026-08-19T20:33:35Z

+ const groundTruth = await example.run(ctx);
+ return { exampleId: example.exampleId, sessionId, groundTruth };
+ } catch (error) {
+ this.logger.debug(


instead of logging and rethrowing, is there a way to enrich the error thrown to avoid noise?

Hweinstock · 2026-08-19T20:34:33Z

+
+ // AgentCore emits spans ~30s-3min after invoke; grade too early and it reads an empty
+ // log group and fails every session. Disabled via SIMULATE_INGESTION_WAIT_MS=0 (tests).
+ const waitMs = Number(process.env.SIMULATE_INGESTION_WAIT_MS ?? 180_000);


is there anything we can poll on instead of a static wait time?

I didn't poll right away because some traces might still be arriving, and I didn't want us to end up with incomplete session data.

how do we know that there aren't traces still coming in after 3 min mark?

Hweinstock · 2026-08-19T20:44:21Z

+
+// A fake AWS layer: control resolves the runtime, data answers each invoke. Records every
+// payload it was asked to send, and per `opts` can fail or delay specific invokes.
+function fakeClients(opts: { fail?: (payload: string) => boolean; delayMs?: number } = {}): {


this feels like a really complex testing setup. Is there a simpler way? I'm wondering if there's a structural change we could make to simplify here.

Can I fixed these tests as a follow up?

Hweinstock · 2026-08-19T20:44:48Z

+ let parsed: unknown;
+ try {
+ parsed = JSON.parse(trimmed);
+ } catch {


should we wire the cause here?

Hweinstock · 2026-08-19T20:47:11Z

+import type { DatasetSchemaType } from "@aws-sdk/client-bedrock-agentcore-control";
+import type { InlineGroundTruth } from "@aws-sdk/client-bedrock-agentcore";
+
+// A record, not a bare string, so a future tool-branching type can widen it by a field.


i feel like the code explains this comment.

Hweinstock

only blocking question is why we do an explicit wait, rest could be polished later/in a follow-up. Is there a way we can be sure the spans we need exist before evaluating or is there a reason we don't have to be?

Hweinstock · 2026-08-24T14:39:15Z

+ throw new InputValidationError("required option '--name <name>' not specified");
+
+ // Ctrl-C aborts the run (invokes, the ingestion wait, the dataset download).
+ const controller = new AbortController();


looks like it was merged, so we should be able to leverage that shared abstraction now.
#1986

Hweinstock · 2026-08-24T14:42:18Z

+import { coreOptsFromCtx } from "../../../utils";
+import { parseRuntimeInvokeHeaders } from "../../../runtime/invoke/request";
+
+// Composes invokeDataset (replay) → startBatchEvaluation (grade). Invoke flags mirror


is the comment describing the implementation of the handler? What value does it provide over the code?

Hweinstock · 2026-08-24T16:13:18Z

+ // Enrich with the example identity so the dropped-invoke reason is self-describing
+ // in firstError, instead of a bare transport message logged separately.
+ const cause = error instanceof Error ? error : new Error(String(error));
+ throw new Error(


to confirm my understanding, this error never hits telemetry because an individual example failing does not bubble up so this error is purely for the user message it creates?

Correct. An individual example failing is caught and dropped by runExamples rather than rethrown, so it never reaches telemetry — it only feeds the user-facing message on a total failure. That was intentional: a run can be hundreds of examples, and I didn't want per-example failures flooding our telemetry.

Hweinstock · 2026-08-24T16:13:49Z

+
+ // AgentCore emits spans ~30s-3min after invoke; grade too early and it reads an empty
+ // log group and fails every session. Disabled via SIMULATE_INGESTION_WAIT_MS=0 (tests).
+ const waitMs = Number(process.env.SIMULATE_INGESTION_WAIT_MS ?? 180_000);


how do we know that there aren't traces still coming in after 3 min mark?

Hweinstock · 2026-08-24T16:17:43Z

+ try {
+ headers.set("Authorization", `Bearer ${bearerToken}`);
+ } catch {
+ throw new InputValidationError("Invalid bearer token");


should we wire the cause up to these errors to get the stack trace? (same with 132, 155 below)

Hweinstock · 2026-08-24T16:18:43Z

+ logger
+ .child({
+ errorName:
+ error instanceof TypeError


is there a reason error instanceof error ? error.name : 'error' doesn't work here?

I'll look into this in the next PR.

Hweinstock · 2026-08-24T16:21:33Z

+};
+
+// InvokeDatasetResult reports the created sessions plus how many examples were invoked
+// vs dropped (a failed invoke is skipped, not fatal). firstError explains a total failure.


does firstError explain a total failure? I thought it stored the first failed example's error?

WE will remove it later. Yes lets do a follow up on this.

Hweinstock · 2026-08-24T16:27:49Z

@@ -0,0 +1,263 @@
+// Disables the post-invoke span-ingestion wait so the replay returns immediately.
+process.env.SIMULATE_INGESTION_WAIT_MS = "0";


would this be a good application of global config? this might help this behavior be more discoverable by other tests that might need the same.

So add to the bunfig.toml to preload a script that run this:
./global-mocks.ts
// Global mocksimport{mock}from"bun:test";// Mock environment variablesprocess.env.SIMULATE_INGESTION_WAIT_MS="0";
bunfig.toml
[test]preload=["./global-mocks.ts"]
ref
Yes I could do that in the follow up Pr.

we should be able to inject the global config down through core with the data we want.

ah that make sense

Hweinstock · 2026-08-24T16:32:38Z

+ return this.groundTruth();
+ }
+
+ // Emit every turn (carrying its prompt), not just those with an expectation: filtering


i'm not quite sure what this comment is saying?

Ill remove it.

Hweinstock · 2026-08-24T21:27:20Z

Is there a way we can be sure the spans we need exist before evaluating or is there a reason we don't have to be?

from chatting offline, sounds like the wait approach is shared by the eval time. Still unsure how they can be confident the evaluator results are not incomplete, but approving since its an existing solution.

…example classes

- move renderJsonTemplate out of shared src/io into core/eval/invokeDataset - invokeRuntime: raw TypeError/Error -> InputValidationError/RuntimeInvokeResponseError - wire error causes in template + dataset JSON parse - invokeDataset: enrich per-example invoke failure instead of log+rethrow - simulate: bubble Ctrl-C cancellation (telemetry) instead of quiet return; clarify --description help; TODO(#1986) shared abort helper - drop type-guaranteed 'no leak' test; keep AbortSignal wiring in composition test - trim stale/redundant comments (runtime.tsx, invokeRuntime DTO note, TurnResult)

…in payload-template Bug bash (real exploratory-account run) surfaced two defects in the shared invokeDataset path: 1. PredefinedExample blind-cast assertions/expected_trajectory as string[] with no load-time validation (unlike turns). A non-array value passed load, burned a live paid invoke, then threw a raw '.map is not a function' mislabeled 'failed to invoke'. Now validated in the constructor -> clean InputValidationError before any invoke. 2. A --payload-template with no {input} placeholder was silently accepted, wasting a full ~3-min replay on a constant payload. Now rejected up front in invokeDataset.

nborges-aws · 2026-08-24T22:40:49Z

+ const sessionId = randomUUID();
+ const ctx: RunContext = {
+ invokeOnce: async (payload) => {
+ const response = await invokeRuntime(


Comparing this to src/handlers/runtime/invoke/request.ts, and I notice that the code calls normalizeRuntimeInvokeRequest to do some validation before invoking. Do you think we need something similar here?

nborges-aws

LGTM. One comment with a question for you

jariy17 · 2026-08-18T22:40:03Z

What

Adds batch-evaluation simulate to replay a dataset against a runtime and grade the resulting sessions.

Each dataset example is a self-running object that owns its own invoker (RunContext). invokeDataset builds one invoker per session and passes it into the example's run(). The example performs the invocation; the machine supplies the invoker and collects the result.

The invoker seam

A dataset example knows what to send and how to turn the responses into ground truth, but not how to reach the runtime. Its RunContext owns that runtime-specific invocation:

// src/core/eval/invokeDataset/example/types.tsexporttypeTurnResult={text: string};exporttypeRunContext={invokeOnce(input: string): Promise<TurnResult>};exportinterfaceExample{readonlyexampleId: string;readonlyschemaType: DatasetSchemaType;// Replay this example against the runtime `ctx` reaches, return neutral ground truth.run(ctx: RunContext): Promise<InlineGroundTruth|undefined>;}

invokeDataset resolves the runtime once, then builds a distinct invoker for each client-generated session and passes it to that example:

constresults=awaitrunExamples(examples,async(example)=>{constctx: RunContext={invokeOnce: async(input)=>{constres=awaitinvokeRuntime(deps,{/* resolved runtime, session, and rendered payload */},options,signal,);return{text: /* drained response body */};},};returnexample.run(ctx);});

The example decides how many times to call invokeOnce and in what order. Its invoker decides how each call reaches the runtime for that session.

Layout

src/core/eval/invokeDataset/
├── load.ts DatasetLoader — pure JSONL parse → shape-classify → new
├── run.ts runExamples — bounded-concurrency pool, failure isolation
├── invokeDataset.test.ts end-to-end golden coverage of the whole path
└── example/
├── types.ts Example interface, RunContext (the invoker), TurnResult
├── predefined.ts PredefinedExample — replays scripted turns, builds ground truth
└── simulated.ts SimulatedExample — not shipped; throws at construction

Plus src/core/invokeRuntime.ts, which extracts runtime invocation from runtime.tsx for reuse by both RuntimeClient and invokeDataset, and src/handlers/eval/batch-evaluation/simulate/index.tsx, which composes invokeDataset with startBatchEvaluation.

Testing

bun run typecheck passes.
bun test: 1544 pass, 0 fail.
invokeDataset.test.ts drives the real EvalClient.invokeDataset over a fake AWS layer with golden fixtures. Its snapshot covers created sessions and inline ground truth across every supported variation while exercising the loader, example classes, concurrency pool, template rendering, and runtime invocation.
simulate.test.tsx snapshots the handler's wrapped sessionMetadata.

codecov-commenter · 2026-08-18T22:42:42Z

Codecov Report

❌ Patch coverage is 96.58444% with 18 lines in your changes missing coverage. Please review.
✅ Project coverage is 97.33%. Comparing base (768ef10) to head (27c810f).

Files with missing lines	Patch %	Lines
src/core/eval.tsx	85.71%	14 Missing ⚠️
src/core/eval/invokeDataset/template.ts	91.30%	2 Missing ⚠️
src/core/eval/invokeDataset/example/simulated.ts	92.30%	1 Missing ⚠️
src/core/invokeRuntime.ts	99.35%	1 Missing ⚠️

Additional details and impacted files

@@ Coverage Diff @@## refactor #2032 +/- ##
============================================
- Coverage 97.36% 97.33% -0.04% 
============================================
Files 410 417 +7 Lines 24796 25181 +385 ============================================
+ Hits 24142 24509 +367 - Misses 654 672 +18

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Hweinstock

didn't get to all the examples work yet, but had a few small comments and a question on how we can simplify testing, because it feels pretty gnarly rn.

Hweinstock · 2026-08-19T19:55:13Z

@@ -0,0 +1,33 @@
+import { InputValidationError } from "../errors";


should we keep this inside evals until there is an opportunity to re-use it? I feel like the io directory should be reserved for shared abstractions.

Yeah, I was thinking the invoke handler could use this but lets leave it in the simulate handler for now.

Hweinstock · 2026-08-19T19:55:23Z

+ try {
+ parsed = JSON.parse(template);
+ } catch {
+ throw new InputValidationError(`--${flagName} must be valid JSON`);


should we wire the cause here?

Hweinstock · 2026-08-19T20:02:10Z

+ qualifier?: string;
+ payloadTemplate: string; // e.g. {"prompt":"{input}"} — {input} is the example's turn input
+ headers?: [string, string][];
+ bearerToken?: string;


q: is there a reason bearerToken is treated different from other headers?

It's the discriminator that selects the auth path, not an application header. Its presence routes the invoke to CUSTOM_JWT path (src/core/invokeRuntime.ts:73)

Hweinstock · 2026-08-19T20:07:33Z

+ flag("dataset-version", "dataset version (with a dataset id)", z.string().optional()),
+ flag("evaluator", "evaluator id(s) to apply", z.array(z.string()).optional()),
+ flag("name", "batch evaluation name (unique in the account)", z.string().optional()),
+ flag("description", "optional description", z.string().optional()),


what exactly is this describing? the simulation itself?

This should be batch-evaluation description. I'll rename this batch-eval-desc, same with name too.

Hweinstock · 2026-08-19T20:08:58Z

+ throw new InputValidationError("required option '--name <name>' not specified");
+
+ // Ctrl-C aborts the run (invokes, the ingestion wait, the dataset download).
+ const controller = new AbortController();


ahhh, theres a shared abstraction for this, but looks like it hasn't been merged yet :(
#1986
maybe we can swap it over as a follow-up once its merged.

looks like it was merged, so we should be able to leverage that shared abstraction now.
#1986

Hweinstock · 2026-08-19T20:33:35Z

+ const groundTruth = await example.run(ctx);
+ return { exampleId: example.exampleId, sessionId, groundTruth };
+ } catch (error) {
+ this.logger.debug(


instead of logging and rethrowing, is there a way to enrich the error thrown to avoid noise?

Hweinstock · 2026-08-19T20:34:33Z

+
+ // AgentCore emits spans ~30s-3min after invoke; grade too early and it reads an empty
+ // log group and fails every session. Disabled via SIMULATE_INGESTION_WAIT_MS=0 (tests).
+ const waitMs = Number(process.env.SIMULATE_INGESTION_WAIT_MS ?? 180_000);


is there anything we can poll on instead of a static wait time?

I didn't poll right away because some traces might still be arriving, and I didn't want us to end up with incomplete session data.

how do we know that there aren't traces still coming in after 3 min mark?

Hweinstock · 2026-08-19T20:44:21Z

+
+// A fake AWS layer: control resolves the runtime, data answers each invoke. Records every
+// payload it was asked to send, and per `opts` can fail or delay specific invokes.
+function fakeClients(opts: { fail?: (payload: string) => boolean; delayMs?: number } = {}): {


this feels like a really complex testing setup. Is there a simpler way? I'm wondering if there's a structural change we could make to simplify here.

Can I fixed these tests as a follow up?

Hweinstock · 2026-08-19T20:44:48Z

+ let parsed: unknown;
+ try {
+ parsed = JSON.parse(trimmed);
+ } catch {


should we wire the cause here?

Hweinstock · 2026-08-19T20:47:11Z

+import type { DatasetSchemaType } from "@aws-sdk/client-bedrock-agentcore-control";
+import type { InlineGroundTruth } from "@aws-sdk/client-bedrock-agentcore";
+
+// A record, not a bare string, so a future tool-branching type can widen it by a field.


i feel like the code explains this comment.

Hweinstock

only blocking question is why we do an explicit wait, rest could be polished later/in a follow-up. Is there a way we can be sure the spans we need exist before evaluating or is there a reason we don't have to be?

Hweinstock · 2026-08-24T14:39:15Z

+ throw new InputValidationError("required option '--name <name>' not specified");
+
+ // Ctrl-C aborts the run (invokes, the ingestion wait, the dataset download).
+ const controller = new AbortController();


looks like it was merged, so we should be able to leverage that shared abstraction now.
#1986

Hweinstock · 2026-08-24T14:42:18Z

+import { coreOptsFromCtx } from "../../../utils";
+import { parseRuntimeInvokeHeaders } from "../../../runtime/invoke/request";
+
+// Composes invokeDataset (replay) → startBatchEvaluation (grade). Invoke flags mirror


is the comment describing the implementation of the handler? What value does it provide over the code?

Hweinstock · 2026-08-24T16:13:18Z

+ // Enrich with the example identity so the dropped-invoke reason is self-describing
+ // in firstError, instead of a bare transport message logged separately.
+ const cause = error instanceof Error ? error : new Error(String(error));
+ throw new Error(


to confirm my understanding, this error never hits telemetry because an individual example failing does not bubble up so this error is purely for the user message it creates?

Correct. An individual example failing is caught and dropped by runExamples rather than rethrown, so it never reaches telemetry — it only feeds the user-facing message on a total failure. That was intentional: a run can be hundreds of examples, and I didn't want per-example failures flooding our telemetry.

Hweinstock · 2026-08-24T16:13:49Z

+
+ // AgentCore emits spans ~30s-3min after invoke; grade too early and it reads an empty
+ // log group and fails every session. Disabled via SIMULATE_INGESTION_WAIT_MS=0 (tests).
+ const waitMs = Number(process.env.SIMULATE_INGESTION_WAIT_MS ?? 180_000);


how do we know that there aren't traces still coming in after 3 min mark?

Hweinstock · 2026-08-24T16:17:43Z

+ try {
+ headers.set("Authorization", `Bearer ${bearerToken}`);
+ } catch {
+ throw new InputValidationError("Invalid bearer token");


should we wire the cause up to these errors to get the stack trace? (same with 132, 155 below)

Hweinstock · 2026-08-24T16:18:43Z

+ logger
+ .child({
+ errorName:
+ error instanceof TypeError


is there a reason error instanceof error ? error.name : 'error' doesn't work here?

I'll look into this in the next PR.

Hweinstock · 2026-08-24T16:21:33Z

+};
+
+// InvokeDatasetResult reports the created sessions plus how many examples were invoked
+// vs dropped (a failed invoke is skipped, not fatal). firstError explains a total failure.


does firstError explain a total failure? I thought it stored the first failed example's error?

WE will remove it later. Yes lets do a follow up on this.

Hweinstock · 2026-08-24T16:27:49Z

@@ -0,0 +1,263 @@
+// Disables the post-invoke span-ingestion wait so the replay returns immediately.
+process.env.SIMULATE_INGESTION_WAIT_MS = "0";


would this be a good application of global config? this might help this behavior be more discoverable by other tests that might need the same.

So add to the bunfig.toml to preload a script that run this:
./global-mocks.ts
// Global mocksimport{mock}from"bun:test";// Mock environment variablesprocess.env.SIMULATE_INGESTION_WAIT_MS="0";
bunfig.toml
[test]preload=["./global-mocks.ts"]
ref
Yes I could do that in the follow up Pr.

we should be able to inject the global config down through core with the data we want.

ah that make sense

Hweinstock · 2026-08-24T16:32:38Z

+ return this.groundTruth();
+ }
+
+ // Emit every turn (carrying its prompt), not just those with an expectation: filtering


i'm not quite sure what this comment is saying?

Ill remove it.

Hweinstock · 2026-08-24T21:27:20Z

Is there a way we can be sure the spans we need exist before evaluating or is there a reason we don't have to be?

from chatting offline, sounds like the wait approach is shared by the eval time. Still unsure how they can be confident the evaluator results are not incomplete, but approving since its an existing solution.

…example classes

- move renderJsonTemplate out of shared src/io into core/eval/invokeDataset - invokeRuntime: raw TypeError/Error -> InputValidationError/RuntimeInvokeResponseError - wire error causes in template + dataset JSON parse - invokeDataset: enrich per-example invoke failure instead of log+rethrow - simulate: bubble Ctrl-C cancellation (telemetry) instead of quiet return; clarify --description help; TODO(#1986) shared abort helper - drop type-guaranteed 'no leak' test; keep AbortSignal wiring in composition test - trim stale/redundant comments (runtime.tsx, invokeRuntime DTO note, TurnResult)

…in payload-template Bug bash (real exploratory-account run) surfaced two defects in the shared invokeDataset path: 1. PredefinedExample blind-cast assertions/expected_trajectory as string[] with no load-time validation (unlike turns). A non-array value passed load, burned a live paid invoke, then threw a raw '.map is not a function' mislabeled 'failed to invoke'. Now validated in the constructor -> clean InputValidationError before any invoke. 2. A --payload-template with no {input} placeholder was silently accepted, wasting a full ~3-min replay on a constant payload. Now rejected up front in invokeDataset.

nborges-aws · 2026-08-24T22:40:49Z

+ const sessionId = randomUUID();
+ const ctx: RunContext = {
+ invokeOnce: async (payload) => {
+ const response = await invokeRuntime(


Comparing this to src/handlers/runtime/invoke/request.ts, and I notice that the code calls normalizeRuntimeInvokeRequest to do some validation before invoking. Do you think we need something similar here?

nborges-aws

LGTM. One comment with a question for you

jariy17 · 2026-08-18T22:40:03Z

What

Adds batch-evaluation simulate to replay a dataset against a runtime and grade the resulting sessions.

Each dataset example is a self-running object that owns its own invoker (RunContext). invokeDataset builds one invoker per session and passes it into the example's run(). The example performs the invocation; the machine supplies the invoker and collects the result.

The invoker seam

A dataset example knows what to send and how to turn the responses into ground truth, but not how to reach the runtime. Its RunContext owns that runtime-specific invocation:

// src/core/eval/invokeDataset/example/types.tsexporttypeTurnResult={text: string};exporttypeRunContext={invokeOnce(input: string): Promise<TurnResult>};exportinterfaceExample{readonlyexampleId: string;readonlyschemaType: DatasetSchemaType;// Replay this example against the runtime `ctx` reaches, return neutral ground truth.run(ctx: RunContext): Promise<InlineGroundTruth|undefined>;}

invokeDataset resolves the runtime once, then builds a distinct invoker for each client-generated session and passes it to that example:

constresults=awaitrunExamples(examples,async(example)=>{constctx: RunContext={invokeOnce: async(input)=>{constres=awaitinvokeRuntime(deps,{/* resolved runtime, session, and rendered payload */},options,signal,);return{text: /* drained response body */};},};returnexample.run(ctx);});

The example decides how many times to call invokeOnce and in what order. Its invoker decides how each call reaches the runtime for that session.

Layout

src/core/eval/invokeDataset/
├── load.ts DatasetLoader — pure JSONL parse → shape-classify → new
├── run.ts runExamples — bounded-concurrency pool, failure isolation
├── invokeDataset.test.ts end-to-end golden coverage of the whole path
└── example/
├── types.ts Example interface, RunContext (the invoker), TurnResult
├── predefined.ts PredefinedExample — replays scripted turns, builds ground truth
└── simulated.ts SimulatedExample — not shipped; throws at construction

Plus src/core/invokeRuntime.ts, which extracts runtime invocation from runtime.tsx for reuse by both RuntimeClient and invokeDataset, and src/handlers/eval/batch-evaluation/simulate/index.tsx, which composes invokeDataset with startBatchEvaluation.

Testing

bun run typecheck passes.
bun test: 1544 pass, 0 fail.
invokeDataset.test.ts drives the real EvalClient.invokeDataset over a fake AWS layer with golden fixtures. Its snapshot covers created sessions and inline ground truth across every supported variation while exercising the loader, example classes, concurrency pool, template rendering, and runtime invocation.
simulate.test.tsx snapshots the handler's wrapped sessionMetadata.

codecov-commenter · 2026-08-18T22:42:42Z

Codecov Report

❌ Patch coverage is 96.58444% with 18 lines in your changes missing coverage. Please review.
✅ Project coverage is 97.33%. Comparing base (768ef10) to head (27c810f).

Files with missing lines	Patch %	Lines
src/core/eval.tsx	85.71%	14 Missing ⚠️
src/core/eval/invokeDataset/template.ts	91.30%	2 Missing ⚠️
src/core/eval/invokeDataset/example/simulated.ts	92.30%	1 Missing ⚠️
src/core/invokeRuntime.ts	99.35%	1 Missing ⚠️

Additional details and impacted files

@@ Coverage Diff @@## refactor #2032 +/- ##
============================================
- Coverage 97.36% 97.33% -0.04% 
============================================
Files 410 417 +7 Lines 24796 25181 +385 ============================================
+ Hits 24142 24509 +367 - Misses 654 672 +18

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Hweinstock

didn't get to all the examples work yet, but had a few small comments and a question on how we can simplify testing, because it feels pretty gnarly rn.

Hweinstock · 2026-08-19T19:55:13Z

@@ -0,0 +1,33 @@
+import { InputValidationError } from "../errors";


should we keep this inside evals until there is an opportunity to re-use it? I feel like the io directory should be reserved for shared abstractions.

Yeah, I was thinking the invoke handler could use this but lets leave it in the simulate handler for now.

Hweinstock · 2026-08-19T19:55:23Z

+ try {
+ parsed = JSON.parse(template);
+ } catch {
+ throw new InputValidationError(`--${flagName} must be valid JSON`);


should we wire the cause here?

Hweinstock · 2026-08-19T20:02:10Z

+ qualifier?: string;
+ payloadTemplate: string; // e.g. {"prompt":"{input}"} — {input} is the example's turn input
+ headers?: [string, string][];
+ bearerToken?: string;


q: is there a reason bearerToken is treated different from other headers?

It's the discriminator that selects the auth path, not an application header. Its presence routes the invoke to CUSTOM_JWT path (src/core/invokeRuntime.ts:73)

Hweinstock · 2026-08-19T20:07:33Z

+ flag("dataset-version", "dataset version (with a dataset id)", z.string().optional()),
+ flag("evaluator", "evaluator id(s) to apply", z.array(z.string()).optional()),
+ flag("name", "batch evaluation name (unique in the account)", z.string().optional()),
+ flag("description", "optional description", z.string().optional()),


what exactly is this describing? the simulation itself?

This should be batch-evaluation description. I'll rename this batch-eval-desc, same with name too.

Hweinstock · 2026-08-19T20:08:58Z

+ throw new InputValidationError("required option '--name <name>' not specified");
+
+ // Ctrl-C aborts the run (invokes, the ingestion wait, the dataset download).
+ const controller = new AbortController();


ahhh, theres a shared abstraction for this, but looks like it hasn't been merged yet :(
#1986
maybe we can swap it over as a follow-up once its merged.

looks like it was merged, so we should be able to leverage that shared abstraction now.
#1986

Hweinstock · 2026-08-19T20:33:35Z

+ const groundTruth = await example.run(ctx);
+ return { exampleId: example.exampleId, sessionId, groundTruth };
+ } catch (error) {
+ this.logger.debug(


instead of logging and rethrowing, is there a way to enrich the error thrown to avoid noise?

Hweinstock · 2026-08-19T20:34:33Z

+
+ // AgentCore emits spans ~30s-3min after invoke; grade too early and it reads an empty
+ // log group and fails every session. Disabled via SIMULATE_INGESTION_WAIT_MS=0 (tests).
+ const waitMs = Number(process.env.SIMULATE_INGESTION_WAIT_MS ?? 180_000);


is there anything we can poll on instead of a static wait time?

I didn't poll right away because some traces might still be arriving, and I didn't want us to end up with incomplete session data.

how do we know that there aren't traces still coming in after 3 min mark?

Hweinstock · 2026-08-19T20:44:21Z

+
+// A fake AWS layer: control resolves the runtime, data answers each invoke. Records every
+// payload it was asked to send, and per `opts` can fail or delay specific invokes.
+function fakeClients(opts: { fail?: (payload: string) => boolean; delayMs?: number } = {}): {


this feels like a really complex testing setup. Is there a simpler way? I'm wondering if there's a structural change we could make to simplify here.

Can I fixed these tests as a follow up?

Hweinstock · 2026-08-19T20:44:48Z

+ let parsed: unknown;
+ try {
+ parsed = JSON.parse(trimmed);
+ } catch {


should we wire the cause here?

Hweinstock · 2026-08-19T20:47:11Z

+import type { DatasetSchemaType } from "@aws-sdk/client-bedrock-agentcore-control";
+import type { InlineGroundTruth } from "@aws-sdk/client-bedrock-agentcore";
+
+// A record, not a bare string, so a future tool-branching type can widen it by a field.


i feel like the code explains this comment.

Hweinstock

only blocking question is why we do an explicit wait, rest could be polished later/in a follow-up. Is there a way we can be sure the spans we need exist before evaluating or is there a reason we don't have to be?

Hweinstock · 2026-08-24T14:39:15Z

+ throw new InputValidationError("required option '--name <name>' not specified");
+
+ // Ctrl-C aborts the run (invokes, the ingestion wait, the dataset download).
+ const controller = new AbortController();


looks like it was merged, so we should be able to leverage that shared abstraction now.
#1986

Hweinstock · 2026-08-24T14:42:18Z

+import { coreOptsFromCtx } from "../../../utils";
+import { parseRuntimeInvokeHeaders } from "../../../runtime/invoke/request";
+
+// Composes invokeDataset (replay) → startBatchEvaluation (grade). Invoke flags mirror


is the comment describing the implementation of the handler? What value does it provide over the code?

Hweinstock · 2026-08-24T16:13:18Z

+ // Enrich with the example identity so the dropped-invoke reason is self-describing
+ // in firstError, instead of a bare transport message logged separately.
+ const cause = error instanceof Error ? error : new Error(String(error));
+ throw new Error(


to confirm my understanding, this error never hits telemetry because an individual example failing does not bubble up so this error is purely for the user message it creates?

Correct. An individual example failing is caught and dropped by runExamples rather than rethrown, so it never reaches telemetry — it only feeds the user-facing message on a total failure. That was intentional: a run can be hundreds of examples, and I didn't want per-example failures flooding our telemetry.

Hweinstock · 2026-08-24T16:13:49Z

+
+ // AgentCore emits spans ~30s-3min after invoke; grade too early and it reads an empty
+ // log group and fails every session. Disabled via SIMULATE_INGESTION_WAIT_MS=0 (tests).
+ const waitMs = Number(process.env.SIMULATE_INGESTION_WAIT_MS ?? 180_000);


how do we know that there aren't traces still coming in after 3 min mark?

Hweinstock · 2026-08-24T16:17:43Z

+ try {
+ headers.set("Authorization", `Bearer ${bearerToken}`);
+ } catch {
+ throw new InputValidationError("Invalid bearer token");


should we wire the cause up to these errors to get the stack trace? (same with 132, 155 below)

Hweinstock · 2026-08-24T16:18:43Z

+ logger
+ .child({
+ errorName:
+ error instanceof TypeError


is there a reason error instanceof error ? error.name : 'error' doesn't work here?

I'll look into this in the next PR.

Hweinstock · 2026-08-24T16:21:33Z

+};
+
+// InvokeDatasetResult reports the created sessions plus how many examples were invoked
+// vs dropped (a failed invoke is skipped, not fatal). firstError explains a total failure.


does firstError explain a total failure? I thought it stored the first failed example's error?

WE will remove it later. Yes lets do a follow up on this.

Hweinstock · 2026-08-24T16:27:49Z

@@ -0,0 +1,263 @@
+// Disables the post-invoke span-ingestion wait so the replay returns immediately.
+process.env.SIMULATE_INGESTION_WAIT_MS = "0";


would this be a good application of global config? this might help this behavior be more discoverable by other tests that might need the same.

So add to the bunfig.toml to preload a script that run this:
./global-mocks.ts
// Global mocksimport{mock}from"bun:test";// Mock environment variablesprocess.env.SIMULATE_INGESTION_WAIT_MS="0";
bunfig.toml
[test]preload=["./global-mocks.ts"]
ref
Yes I could do that in the follow up Pr.

we should be able to inject the global config down through core with the data we want.

ah that make sense

Hweinstock · 2026-08-24T16:32:38Z

+ return this.groundTruth();
+ }
+
+ // Emit every turn (carrying its prompt), not just those with an expectation: filtering


i'm not quite sure what this comment is saying?

Ill remove it.

Hweinstock · 2026-08-24T21:27:20Z

Is there a way we can be sure the spans we need exist before evaluating or is there a reason we don't have to be?

from chatting offline, sounds like the wait approach is shared by the eval time. Still unsure how they can be confident the evaluator results are not incomplete, but approving since its an existing solution.

…example classes

- move renderJsonTemplate out of shared src/io into core/eval/invokeDataset - invokeRuntime: raw TypeError/Error -> InputValidationError/RuntimeInvokeResponseError - wire error causes in template + dataset JSON parse - invokeDataset: enrich per-example invoke failure instead of log+rethrow - simulate: bubble Ctrl-C cancellation (telemetry) instead of quiet return; clarify --description help; TODO(#1986) shared abort helper - drop type-guaranteed 'no leak' test; keep AbortSignal wiring in composition test - trim stale/redundant comments (runtime.tsx, invokeRuntime DTO note, TurnResult)

…in payload-template Bug bash (real exploratory-account run) surfaced two defects in the shared invokeDataset path: 1. PredefinedExample blind-cast assertions/expected_trajectory as string[] with no load-time validation (unlike turns). A non-array value passed load, burned a live paid invoke, then threw a raw '.map is not a function' mislabeled 'failed to invoke'. Now validated in the constructor -> clean InputValidationError before any invoke. 2. A --payload-template with no {input} placeholder was silently accepted, wasting a full ~3-min replay on a constant payload. Now rejected up front in invokeDataset.

nborges-aws · 2026-08-24T22:40:49Z

+ const sessionId = randomUUID();
+ const ctx: RunContext = {
+ invokeOnce: async (payload) => {
+ const response = await invokeRuntime(


Comparing this to src/handlers/runtime/invoke/request.ts, and I notice that the code calls normalizeRuntimeInvokeRequest to do some validation before invoking. Do you think we need something similar here?

nborges-aws

LGTM. One comment with a question for you

jariy17 · 2026-08-18T22:40:03Z

What

Adds batch-evaluation simulate to replay a dataset against a runtime and grade the resulting sessions.

Each dataset example is a self-running object that owns its own invoker (RunContext). invokeDataset builds one invoker per session and passes it into the example's run(). The example performs the invocation; the machine supplies the invoker and collects the result.

The invoker seam

A dataset example knows what to send and how to turn the responses into ground truth, but not how to reach the runtime. Its RunContext owns that runtime-specific invocation:

// src/core/eval/invokeDataset/example/types.tsexporttypeTurnResult={text: string};exporttypeRunContext={invokeOnce(input: string): Promise<TurnResult>};exportinterfaceExample{readonlyexampleId: string;readonlyschemaType: DatasetSchemaType;// Replay this example against the runtime `ctx` reaches, return neutral ground truth.run(ctx: RunContext): Promise<InlineGroundTruth|undefined>;}

invokeDataset resolves the runtime once, then builds a distinct invoker for each client-generated session and passes it to that example:

constresults=awaitrunExamples(examples,async(example)=>{constctx: RunContext={invokeOnce: async(input)=>{constres=awaitinvokeRuntime(deps,{/* resolved runtime, session, and rendered payload */},options,signal,);return{text: /* drained response body */};},};returnexample.run(ctx);});

The example decides how many times to call invokeOnce and in what order. Its invoker decides how each call reaches the runtime for that session.

Layout

src/core/eval/invokeDataset/
├── load.ts DatasetLoader — pure JSONL parse → shape-classify → new
├── run.ts runExamples — bounded-concurrency pool, failure isolation
├── invokeDataset.test.ts end-to-end golden coverage of the whole path
└── example/
├── types.ts Example interface, RunContext (the invoker), TurnResult
├── predefined.ts PredefinedExample — replays scripted turns, builds ground truth
└── simulated.ts SimulatedExample — not shipped; throws at construction

Plus src/core/invokeRuntime.ts, which extracts runtime invocation from runtime.tsx for reuse by both RuntimeClient and invokeDataset, and src/handlers/eval/batch-evaluation/simulate/index.tsx, which composes invokeDataset with startBatchEvaluation.

Testing

bun run typecheck passes.
bun test: 1544 pass, 0 fail.
invokeDataset.test.ts drives the real EvalClient.invokeDataset over a fake AWS layer with golden fixtures. Its snapshot covers created sessions and inline ground truth across every supported variation while exercising the loader, example classes, concurrency pool, template rendering, and runtime invocation.
simulate.test.tsx snapshots the handler's wrapped sessionMetadata.

codecov-commenter · 2026-08-18T22:42:42Z

Codecov Report

❌ Patch coverage is 96.58444% with 18 lines in your changes missing coverage. Please review.
✅ Project coverage is 97.33%. Comparing base (768ef10) to head (27c810f).

Files with missing lines	Patch %	Lines
src/core/eval.tsx	85.71%	14 Missing ⚠️
src/core/eval/invokeDataset/template.ts	91.30%	2 Missing ⚠️
src/core/eval/invokeDataset/example/simulated.ts	92.30%	1 Missing ⚠️
src/core/invokeRuntime.ts	99.35%	1 Missing ⚠️

Additional details and impacted files

@@ Coverage Diff @@## refactor #2032 +/- ##
============================================
- Coverage 97.36% 97.33% -0.04% 
============================================
Files 410 417 +7 Lines 24796 25181 +385 ============================================
+ Hits 24142 24509 +367 - Misses 654 672 +18

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Hweinstock

didn't get to all the examples work yet, but had a few small comments and a question on how we can simplify testing, because it feels pretty gnarly rn.

Hweinstock · 2026-08-19T19:55:13Z

@@ -0,0 +1,33 @@
+import { InputValidationError } from "../errors";


should we keep this inside evals until there is an opportunity to re-use it? I feel like the io directory should be reserved for shared abstractions.

Yeah, I was thinking the invoke handler could use this but lets leave it in the simulate handler for now.

Hweinstock · 2026-08-19T19:55:23Z

+ try {
+ parsed = JSON.parse(template);
+ } catch {
+ throw new InputValidationError(`--${flagName} must be valid JSON`);


should we wire the cause here?

Hweinstock · 2026-08-19T20:02:10Z

+ qualifier?: string;
+ payloadTemplate: string; // e.g. {"prompt":"{input}"} — {input} is the example's turn input
+ headers?: [string, string][];
+ bearerToken?: string;


q: is there a reason bearerToken is treated different from other headers?

It's the discriminator that selects the auth path, not an application header. Its presence routes the invoke to CUSTOM_JWT path (src/core/invokeRuntime.ts:73)

Hweinstock · 2026-08-19T20:07:33Z

+ flag("dataset-version", "dataset version (with a dataset id)", z.string().optional()),
+ flag("evaluator", "evaluator id(s) to apply", z.array(z.string()).optional()),
+ flag("name", "batch evaluation name (unique in the account)", z.string().optional()),
+ flag("description", "optional description", z.string().optional()),


what exactly is this describing? the simulation itself?

This should be batch-evaluation description. I'll rename this batch-eval-desc, same with name too.

Hweinstock · 2026-08-19T20:08:58Z

+ throw new InputValidationError("required option '--name <name>' not specified");
+
+ // Ctrl-C aborts the run (invokes, the ingestion wait, the dataset download).
+ const controller = new AbortController();


ahhh, theres a shared abstraction for this, but looks like it hasn't been merged yet :(
#1986
maybe we can swap it over as a follow-up once its merged.

looks like it was merged, so we should be able to leverage that shared abstraction now.
#1986

Hweinstock · 2026-08-19T20:33:35Z

+ const groundTruth = await example.run(ctx);
+ return { exampleId: example.exampleId, sessionId, groundTruth };
+ } catch (error) {
+ this.logger.debug(


instead of logging and rethrowing, is there a way to enrich the error thrown to avoid noise?

Hweinstock · 2026-08-19T20:34:33Z

+
+ // AgentCore emits spans ~30s-3min after invoke; grade too early and it reads an empty
+ // log group and fails every session. Disabled via SIMULATE_INGESTION_WAIT_MS=0 (tests).
+ const waitMs = Number(process.env.SIMULATE_INGESTION_WAIT_MS ?? 180_000);


is there anything we can poll on instead of a static wait time?

I didn't poll right away because some traces might still be arriving, and I didn't want us to end up with incomplete session data.

how do we know that there aren't traces still coming in after 3 min mark?

Hweinstock · 2026-08-19T20:44:21Z

+
+// A fake AWS layer: control resolves the runtime, data answers each invoke. Records every
+// payload it was asked to send, and per `opts` can fail or delay specific invokes.
+function fakeClients(opts: { fail?: (payload: string) => boolean; delayMs?: number } = {}): {


this feels like a really complex testing setup. Is there a simpler way? I'm wondering if there's a structural change we could make to simplify here.

Can I fixed these tests as a follow up?

Hweinstock · 2026-08-19T20:44:48Z

+ let parsed: unknown;
+ try {
+ parsed = JSON.parse(trimmed);
+ } catch {


should we wire the cause here?

Hweinstock · 2026-08-19T20:47:11Z

+import type { DatasetSchemaType } from "@aws-sdk/client-bedrock-agentcore-control";
+import type { InlineGroundTruth } from "@aws-sdk/client-bedrock-agentcore";
+
+// A record, not a bare string, so a future tool-branching type can widen it by a field.


i feel like the code explains this comment.

Hweinstock

only blocking question is why we do an explicit wait, rest could be polished later/in a follow-up. Is there a way we can be sure the spans we need exist before evaluating or is there a reason we don't have to be?

Hweinstock · 2026-08-24T14:39:15Z

+ throw new InputValidationError("required option '--name <name>' not specified");
+
+ // Ctrl-C aborts the run (invokes, the ingestion wait, the dataset download).
+ const controller = new AbortController();


looks like it was merged, so we should be able to leverage that shared abstraction now.
#1986

Hweinstock · 2026-08-24T14:42:18Z

+import { coreOptsFromCtx } from "../../../utils";
+import { parseRuntimeInvokeHeaders } from "../../../runtime/invoke/request";
+
+// Composes invokeDataset (replay) → startBatchEvaluation (grade). Invoke flags mirror


is the comment describing the implementation of the handler? What value does it provide over the code?

Hweinstock · 2026-08-24T16:13:18Z

+ // Enrich with the example identity so the dropped-invoke reason is self-describing
+ // in firstError, instead of a bare transport message logged separately.
+ const cause = error instanceof Error ? error : new Error(String(error));
+ throw new Error(


to confirm my understanding, this error never hits telemetry because an individual example failing does not bubble up so this error is purely for the user message it creates?

Correct. An individual example failing is caught and dropped by runExamples rather than rethrown, so it never reaches telemetry — it only feeds the user-facing message on a total failure. That was intentional: a run can be hundreds of examples, and I didn't want per-example failures flooding our telemetry.

Hweinstock · 2026-08-24T16:13:49Z

+
+ // AgentCore emits spans ~30s-3min after invoke; grade too early and it reads an empty
+ // log group and fails every session. Disabled via SIMULATE_INGESTION_WAIT_MS=0 (tests).
+ const waitMs = Number(process.env.SIMULATE_INGESTION_WAIT_MS ?? 180_000);


how do we know that there aren't traces still coming in after 3 min mark?

Hweinstock · 2026-08-24T16:17:43Z

+ try {
+ headers.set("Authorization", `Bearer ${bearerToken}`);
+ } catch {
+ throw new InputValidationError("Invalid bearer token");


should we wire the cause up to these errors to get the stack trace? (same with 132, 155 below)

Hweinstock · 2026-08-24T16:18:43Z

+ logger
+ .child({
+ errorName:
+ error instanceof TypeError


is there a reason error instanceof error ? error.name : 'error' doesn't work here?

I'll look into this in the next PR.

Hweinstock · 2026-08-24T16:21:33Z

+};
+
+// InvokeDatasetResult reports the created sessions plus how many examples were invoked
+// vs dropped (a failed invoke is skipped, not fatal). firstError explains a total failure.


does firstError explain a total failure? I thought it stored the first failed example's error?

WE will remove it later. Yes lets do a follow up on this.

Hweinstock · 2026-08-24T16:27:49Z

@@ -0,0 +1,263 @@
+// Disables the post-invoke span-ingestion wait so the replay returns immediately.
+process.env.SIMULATE_INGESTION_WAIT_MS = "0";


would this be a good application of global config? this might help this behavior be more discoverable by other tests that might need the same.

So add to the bunfig.toml to preload a script that run this:
./global-mocks.ts
// Global mocksimport{mock}from"bun:test";// Mock environment variablesprocess.env.SIMULATE_INGESTION_WAIT_MS="0";
bunfig.toml
[test]preload=["./global-mocks.ts"]
ref
Yes I could do that in the follow up Pr.

we should be able to inject the global config down through core with the data we want.

ah that make sense

Hweinstock · 2026-08-24T16:32:38Z

+ return this.groundTruth();
+ }
+
+ // Emit every turn (carrying its prompt), not just those with an expectation: filtering


i'm not quite sure what this comment is saying?

Ill remove it.

Hweinstock · 2026-08-24T21:27:20Z

Is there a way we can be sure the spans we need exist before evaluating or is there a reason we don't have to be?

from chatting offline, sounds like the wait approach is shared by the eval time. Still unsure how they can be confident the evaluator results are not incomplete, but approving since its an existing solution.

…example classes

- move renderJsonTemplate out of shared src/io into core/eval/invokeDataset - invokeRuntime: raw TypeError/Error -> InputValidationError/RuntimeInvokeResponseError - wire error causes in template + dataset JSON parse - invokeDataset: enrich per-example invoke failure instead of log+rethrow - simulate: bubble Ctrl-C cancellation (telemetry) instead of quiet return; clarify --description help; TODO(#1986) shared abort helper - drop type-guaranteed 'no leak' test; keep AbortSignal wiring in composition test - trim stale/redundant comments (runtime.tsx, invokeRuntime DTO note, TurnResult)

…in payload-template Bug bash (real exploratory-account run) surfaced two defects in the shared invokeDataset path: 1. PredefinedExample blind-cast assertions/expected_trajectory as string[] with no load-time validation (unlike turns). A non-array value passed load, burned a live paid invoke, then threw a raw '.map is not a function' mislabeled 'failed to invoke'. Now validated in the constructor -> clean InputValidationError before any invoke. 2. A --payload-template with no {input} placeholder was silently accepted, wasting a full ~3-min replay on a constant payload. Now rejected up front in invokeDataset.

nborges-aws · 2026-08-24T22:40:49Z

+ const sessionId = randomUUID();
+ const ctx: RunContext = {
+ invokeOnce: async (payload) => {
+ const response = await invokeRuntime(


Comparing this to src/handlers/runtime/invoke/request.ts, and I notice that the code calls normalizeRuntimeInvokeRequest to do some validation before invoking. Do you think we need something similar here?

nborges-aws

LGTM. One comment with a question for you

jariy17 · 2026-08-18T22:40:03Z

What

Adds batch-evaluation simulate to replay a dataset against a runtime and grade the resulting sessions.

Each dataset example is a self-running object that owns its own invoker (RunContext). invokeDataset builds one invoker per session and passes it into the example's run(). The example performs the invocation; the machine supplies the invoker and collects the result.

The invoker seam

A dataset example knows what to send and how to turn the responses into ground truth, but not how to reach the runtime. Its RunContext owns that runtime-specific invocation:

// src/core/eval/invokeDataset/example/types.tsexporttypeTurnResult={text: string};exporttypeRunContext={invokeOnce(input: string): Promise<TurnResult>};exportinterfaceExample{readonlyexampleId: string;readonlyschemaType: DatasetSchemaType;// Replay this example against the runtime `ctx` reaches, return neutral ground truth.run(ctx: RunContext): Promise<InlineGroundTruth|undefined>;}

invokeDataset resolves the runtime once, then builds a distinct invoker for each client-generated session and passes it to that example:

constresults=awaitrunExamples(examples,async(example)=>{constctx: RunContext={invokeOnce: async(input)=>{constres=awaitinvokeRuntime(deps,{/* resolved runtime, session, and rendered payload */},options,signal,);return{text: /* drained response body */};},};returnexample.run(ctx);});

The example decides how many times to call invokeOnce and in what order. Its invoker decides how each call reaches the runtime for that session.

Layout

src/core/eval/invokeDataset/
├── load.ts DatasetLoader — pure JSONL parse → shape-classify → new
├── run.ts runExamples — bounded-concurrency pool, failure isolation
├── invokeDataset.test.ts end-to-end golden coverage of the whole path
└── example/
├── types.ts Example interface, RunContext (the invoker), TurnResult
├── predefined.ts PredefinedExample — replays scripted turns, builds ground truth
└── simulated.ts SimulatedExample — not shipped; throws at construction

Plus src/core/invokeRuntime.ts, which extracts runtime invocation from runtime.tsx for reuse by both RuntimeClient and invokeDataset, and src/handlers/eval/batch-evaluation/simulate/index.tsx, which composes invokeDataset with startBatchEvaluation.

Testing

bun run typecheck passes.
bun test: 1544 pass, 0 fail.
invokeDataset.test.ts drives the real EvalClient.invokeDataset over a fake AWS layer with golden fixtures. Its snapshot covers created sessions and inline ground truth across every supported variation while exercising the loader, example classes, concurrency pool, template rendering, and runtime invocation.
simulate.test.tsx snapshots the handler's wrapped sessionMetadata.

codecov-commenter · 2026-08-18T22:42:42Z

Codecov Report

❌ Patch coverage is 96.58444% with 18 lines in your changes missing coverage. Please review.
✅ Project coverage is 97.33%. Comparing base (768ef10) to head (27c810f).

Files with missing lines	Patch %	Lines
src/core/eval.tsx	85.71%	14 Missing ⚠️
src/core/eval/invokeDataset/template.ts	91.30%	2 Missing ⚠️
src/core/eval/invokeDataset/example/simulated.ts	92.30%	1 Missing ⚠️
src/core/invokeRuntime.ts	99.35%	1 Missing ⚠️

Additional details and impacted files

@@ Coverage Diff @@## refactor #2032 +/- ##
============================================
- Coverage 97.36% 97.33% -0.04% 
============================================
Files 410 417 +7 Lines 24796 25181 +385 ============================================
+ Hits 24142 24509 +367 - Misses 654 672 +18

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Hweinstock

didn't get to all the examples work yet, but had a few small comments and a question on how we can simplify testing, because it feels pretty gnarly rn.

Hweinstock · 2026-08-19T19:55:13Z

@@ -0,0 +1,33 @@
+import { InputValidationError } from "../errors";


should we keep this inside evals until there is an opportunity to re-use it? I feel like the io directory should be reserved for shared abstractions.

Yeah, I was thinking the invoke handler could use this but lets leave it in the simulate handler for now.

Hweinstock · 2026-08-19T19:55:23Z

+ try {
+ parsed = JSON.parse(template);
+ } catch {
+ throw new InputValidationError(`--${flagName} must be valid JSON`);


should we wire the cause here?

Hweinstock · 2026-08-19T20:02:10Z

+ qualifier?: string;
+ payloadTemplate: string; // e.g. {"prompt":"{input}"} — {input} is the example's turn input
+ headers?: [string, string][];
+ bearerToken?: string;


q: is there a reason bearerToken is treated different from other headers?

It's the discriminator that selects the auth path, not an application header. Its presence routes the invoke to CUSTOM_JWT path (src/core/invokeRuntime.ts:73)

Hweinstock · 2026-08-19T20:07:33Z

+ flag("dataset-version", "dataset version (with a dataset id)", z.string().optional()),
+ flag("evaluator", "evaluator id(s) to apply", z.array(z.string()).optional()),
+ flag("name", "batch evaluation name (unique in the account)", z.string().optional()),
+ flag("description", "optional description", z.string().optional()),


what exactly is this describing? the simulation itself?

This should be batch-evaluation description. I'll rename this batch-eval-desc, same with name too.

Hweinstock · 2026-08-19T20:08:58Z

+ throw new InputValidationError("required option '--name <name>' not specified");
+
+ // Ctrl-C aborts the run (invokes, the ingestion wait, the dataset download).
+ const controller = new AbortController();


ahhh, theres a shared abstraction for this, but looks like it hasn't been merged yet :(
#1986
maybe we can swap it over as a follow-up once its merged.

looks like it was merged, so we should be able to leverage that shared abstraction now.
#1986

Hweinstock · 2026-08-19T20:33:35Z

+ const groundTruth = await example.run(ctx);
+ return { exampleId: example.exampleId, sessionId, groundTruth };
+ } catch (error) {
+ this.logger.debug(


instead of logging and rethrowing, is there a way to enrich the error thrown to avoid noise?

Hweinstock · 2026-08-19T20:34:33Z

+
+ // AgentCore emits spans ~30s-3min after invoke; grade too early and it reads an empty
+ // log group and fails every session. Disabled via SIMULATE_INGESTION_WAIT_MS=0 (tests).
+ const waitMs = Number(process.env.SIMULATE_INGESTION_WAIT_MS ?? 180_000);


is there anything we can poll on instead of a static wait time?

I didn't poll right away because some traces might still be arriving, and I didn't want us to end up with incomplete session data.

how do we know that there aren't traces still coming in after 3 min mark?

Hweinstock · 2026-08-19T20:44:21Z

+
+// A fake AWS layer: control resolves the runtime, data answers each invoke. Records every
+// payload it was asked to send, and per `opts` can fail or delay specific invokes.
+function fakeClients(opts: { fail?: (payload: string) => boolean; delayMs?: number } = {}): {


this feels like a really complex testing setup. Is there a simpler way? I'm wondering if there's a structural change we could make to simplify here.

Can I fixed these tests as a follow up?

Hweinstock · 2026-08-19T20:44:48Z

+ let parsed: unknown;
+ try {
+ parsed = JSON.parse(trimmed);
+ } catch {


should we wire the cause here?

Hweinstock · 2026-08-19T20:47:11Z

+import type { DatasetSchemaType } from "@aws-sdk/client-bedrock-agentcore-control";
+import type { InlineGroundTruth } from "@aws-sdk/client-bedrock-agentcore";
+
+// A record, not a bare string, so a future tool-branching type can widen it by a field.


i feel like the code explains this comment.

Hweinstock

only blocking question is why we do an explicit wait, rest could be polished later/in a follow-up. Is there a way we can be sure the spans we need exist before evaluating or is there a reason we don't have to be?

Hweinstock · 2026-08-24T14:39:15Z

+ throw new InputValidationError("required option '--name <name>' not specified");
+
+ // Ctrl-C aborts the run (invokes, the ingestion wait, the dataset download).
+ const controller = new AbortController();


looks like it was merged, so we should be able to leverage that shared abstraction now.
#1986

Hweinstock · 2026-08-24T14:42:18Z

+import { coreOptsFromCtx } from "../../../utils";
+import { parseRuntimeInvokeHeaders } from "../../../runtime/invoke/request";
+
+// Composes invokeDataset (replay) → startBatchEvaluation (grade). Invoke flags mirror


is the comment describing the implementation of the handler? What value does it provide over the code?

Hweinstock · 2026-08-24T16:13:18Z

+ // Enrich with the example identity so the dropped-invoke reason is self-describing
+ // in firstError, instead of a bare transport message logged separately.
+ const cause = error instanceof Error ? error : new Error(String(error));
+ throw new Error(


to confirm my understanding, this error never hits telemetry because an individual example failing does not bubble up so this error is purely for the user message it creates?

Correct. An individual example failing is caught and dropped by runExamples rather than rethrown, so it never reaches telemetry — it only feeds the user-facing message on a total failure. That was intentional: a run can be hundreds of examples, and I didn't want per-example failures flooding our telemetry.

Hweinstock · 2026-08-24T16:13:49Z

+
+ // AgentCore emits spans ~30s-3min after invoke; grade too early and it reads an empty
+ // log group and fails every session. Disabled via SIMULATE_INGESTION_WAIT_MS=0 (tests).
+ const waitMs = Number(process.env.SIMULATE_INGESTION_WAIT_MS ?? 180_000);


how do we know that there aren't traces still coming in after 3 min mark?

Hweinstock · 2026-08-24T16:17:43Z

+ try {
+ headers.set("Authorization", `Bearer ${bearerToken}`);
+ } catch {
+ throw new InputValidationError("Invalid bearer token");


should we wire the cause up to these errors to get the stack trace? (same with 132, 155 below)

Hweinstock · 2026-08-24T16:18:43Z

+ logger
+ .child({
+ errorName:
+ error instanceof TypeError


is there a reason error instanceof error ? error.name : 'error' doesn't work here?

I'll look into this in the next PR.

Hweinstock · 2026-08-24T16:21:33Z

+};
+
+// InvokeDatasetResult reports the created sessions plus how many examples were invoked
+// vs dropped (a failed invoke is skipped, not fatal). firstError explains a total failure.


does firstError explain a total failure? I thought it stored the first failed example's error?

WE will remove it later. Yes lets do a follow up on this.

Hweinstock · 2026-08-24T16:27:49Z

@@ -0,0 +1,263 @@
+// Disables the post-invoke span-ingestion wait so the replay returns immediately.
+process.env.SIMULATE_INGESTION_WAIT_MS = "0";


would this be a good application of global config? this might help this behavior be more discoverable by other tests that might need the same.

So add to the bunfig.toml to preload a script that run this:
./global-mocks.ts
// Global mocksimport{mock}from"bun:test";// Mock environment variablesprocess.env.SIMULATE_INGESTION_WAIT_MS="0";
bunfig.toml
[test]preload=["./global-mocks.ts"]
ref
Yes I could do that in the follow up Pr.

we should be able to inject the global config down through core with the data we want.

ah that make sense

Hweinstock · 2026-08-24T16:32:38Z

+ return this.groundTruth();
+ }
+
+ // Emit every turn (carrying its prompt), not just those with an expectation: filtering


i'm not quite sure what this comment is saying?

Ill remove it.

Hweinstock · 2026-08-24T21:27:20Z

Is there a way we can be sure the spans we need exist before evaluating or is there a reason we don't have to be?

from chatting offline, sounds like the wait approach is shared by the eval time. Still unsure how they can be confident the evaluator results are not incomplete, but approving since its an existing solution.

…example classes

- move renderJsonTemplate out of shared src/io into core/eval/invokeDataset - invokeRuntime: raw TypeError/Error -> InputValidationError/RuntimeInvokeResponseError - wire error causes in template + dataset JSON parse - invokeDataset: enrich per-example invoke failure instead of log+rethrow - simulate: bubble Ctrl-C cancellation (telemetry) instead of quiet return; clarify --description help; TODO(#1986) shared abort helper - drop type-guaranteed 'no leak' test; keep AbortSignal wiring in composition test - trim stale/redundant comments (runtime.tsx, invokeRuntime DTO note, TurnResult)

…in payload-template Bug bash (real exploratory-account run) surfaced two defects in the shared invokeDataset path: 1. PredefinedExample blind-cast assertions/expected_trajectory as string[] with no load-time validation (unlike turns). A non-array value passed load, burned a live paid invoke, then threw a raw '.map is not a function' mislabeled 'failed to invoke'. Now validated in the constructor -> clean InputValidationError before any invoke. 2. A --payload-template with no {input} placeholder was silently accepted, wasting a full ~3-min replay on a constant payload. Now rejected up front in invokeDataset.

nborges-aws · 2026-08-24T22:40:49Z

+ const sessionId = randomUUID();
+ const ctx: RunContext = {
+ invokeOnce: async (payload) => {
+ const response = await invokeRuntime(


Comparing this to src/handlers/runtime/invoke/request.ts, and I notice that the code calls normalizeRuntimeInvokeRequest to do some validation before invoking. Do you think we need something similar here?

nborges-aws

LGTM. One comment with a question for you

jariy17 · 2026-08-18T22:40:03Z

What

Adds batch-evaluation simulate to replay a dataset against a runtime and grade the resulting sessions.

Each dataset example is a self-running object that owns its own invoker (RunContext). invokeDataset builds one invoker per session and passes it into the example's run(). The example performs the invocation; the machine supplies the invoker and collects the result.

The invoker seam

A dataset example knows what to send and how to turn the responses into ground truth, but not how to reach the runtime. Its RunContext owns that runtime-specific invocation:

// src/core/eval/invokeDataset/example/types.tsexporttypeTurnResult={text: string};exporttypeRunContext={invokeOnce(input: string): Promise<TurnResult>};exportinterfaceExample{readonlyexampleId: string;readonlyschemaType: DatasetSchemaType;// Replay this example against the runtime `ctx` reaches, return neutral ground truth.run(ctx: RunContext): Promise<InlineGroundTruth|undefined>;}

invokeDataset resolves the runtime once, then builds a distinct invoker for each client-generated session and passes it to that example:

constresults=awaitrunExamples(examples,async(example)=>{constctx: RunContext={invokeOnce: async(input)=>{constres=awaitinvokeRuntime(deps,{/* resolved runtime, session, and rendered payload */},options,signal,);return{text: /* drained response body */};},};returnexample.run(ctx);});

The example decides how many times to call invokeOnce and in what order. Its invoker decides how each call reaches the runtime for that session.

Layout

src/core/eval/invokeDataset/
├── load.ts DatasetLoader — pure JSONL parse → shape-classify → new
├── run.ts runExamples — bounded-concurrency pool, failure isolation
├── invokeDataset.test.ts end-to-end golden coverage of the whole path
└── example/
├── types.ts Example interface, RunContext (the invoker), TurnResult
├── predefined.ts PredefinedExample — replays scripted turns, builds ground truth
└── simulated.ts SimulatedExample — not shipped; throws at construction

Plus src/core/invokeRuntime.ts, which extracts runtime invocation from runtime.tsx for reuse by both RuntimeClient and invokeDataset, and src/handlers/eval/batch-evaluation/simulate/index.tsx, which composes invokeDataset with startBatchEvaluation.

Testing

bun run typecheck passes.
bun test: 1544 pass, 0 fail.
invokeDataset.test.ts drives the real EvalClient.invokeDataset over a fake AWS layer with golden fixtures. Its snapshot covers created sessions and inline ground truth across every supported variation while exercising the loader, example classes, concurrency pool, template rendering, and runtime invocation.
simulate.test.tsx snapshots the handler's wrapped sessionMetadata.

codecov-commenter · 2026-08-18T22:42:42Z

Codecov Report

❌ Patch coverage is 96.58444% with 18 lines in your changes missing coverage. Please review.
✅ Project coverage is 97.33%. Comparing base (768ef10) to head (27c810f).

Files with missing lines	Patch %	Lines
src/core/eval.tsx	85.71%	14 Missing ⚠️
src/core/eval/invokeDataset/template.ts	91.30%	2 Missing ⚠️
src/core/eval/invokeDataset/example/simulated.ts	92.30%	1 Missing ⚠️
src/core/invokeRuntime.ts	99.35%	1 Missing ⚠️

Additional details and impacted files

@@ Coverage Diff @@## refactor #2032 +/- ##
============================================
- Coverage 97.36% 97.33% -0.04% 
============================================
Files 410 417 +7 Lines 24796 25181 +385 ============================================
+ Hits 24142 24509 +367 - Misses 654 672 +18

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Hweinstock

didn't get to all the examples work yet, but had a few small comments and a question on how we can simplify testing, because it feels pretty gnarly rn.

Hweinstock · 2026-08-19T19:55:13Z

@@ -0,0 +1,33 @@
+import { InputValidationError } from "../errors";


should we keep this inside evals until there is an opportunity to re-use it? I feel like the io directory should be reserved for shared abstractions.

Yeah, I was thinking the invoke handler could use this but lets leave it in the simulate handler for now.

Hweinstock · 2026-08-19T19:55:23Z

+ try {
+ parsed = JSON.parse(template);
+ } catch {
+ throw new InputValidationError(`--${flagName} must be valid JSON`);


should we wire the cause here?

Hweinstock · 2026-08-19T20:02:10Z

+ qualifier?: string;
+ payloadTemplate: string; // e.g. {"prompt":"{input}"} — {input} is the example's turn input
+ headers?: [string, string][];
+ bearerToken?: string;


q: is there a reason bearerToken is treated different from other headers?

It's the discriminator that selects the auth path, not an application header. Its presence routes the invoke to CUSTOM_JWT path (src/core/invokeRuntime.ts:73)

Hweinstock · 2026-08-19T20:07:33Z

+ flag("dataset-version", "dataset version (with a dataset id)", z.string().optional()),
+ flag("evaluator", "evaluator id(s) to apply", z.array(z.string()).optional()),
+ flag("name", "batch evaluation name (unique in the account)", z.string().optional()),
+ flag("description", "optional description", z.string().optional()),


what exactly is this describing? the simulation itself?

This should be batch-evaluation description. I'll rename this batch-eval-desc, same with name too.

Hweinstock · 2026-08-19T20:08:58Z

+ throw new InputValidationError("required option '--name <name>' not specified");
+
+ // Ctrl-C aborts the run (invokes, the ingestion wait, the dataset download).
+ const controller = new AbortController();


ahhh, theres a shared abstraction for this, but looks like it hasn't been merged yet :(
#1986
maybe we can swap it over as a follow-up once its merged.

looks like it was merged, so we should be able to leverage that shared abstraction now.
#1986

Hweinstock · 2026-08-19T20:33:35Z

+ const groundTruth = await example.run(ctx);
+ return { exampleId: example.exampleId, sessionId, groundTruth };
+ } catch (error) {
+ this.logger.debug(


instead of logging and rethrowing, is there a way to enrich the error thrown to avoid noise?

Hweinstock · 2026-08-19T20:34:33Z

+
+ // AgentCore emits spans ~30s-3min after invoke; grade too early and it reads an empty
+ // log group and fails every session. Disabled via SIMULATE_INGESTION_WAIT_MS=0 (tests).
+ const waitMs = Number(process.env.SIMULATE_INGESTION_WAIT_MS ?? 180_000);


is there anything we can poll on instead of a static wait time?

I didn't poll right away because some traces might still be arriving, and I didn't want us to end up with incomplete session data.

how do we know that there aren't traces still coming in after 3 min mark?

Hweinstock · 2026-08-19T20:44:21Z

+
+// A fake AWS layer: control resolves the runtime, data answers each invoke. Records every
+// payload it was asked to send, and per `opts` can fail or delay specific invokes.
+function fakeClients(opts: { fail?: (payload: string) => boolean; delayMs?: number } = {}): {


this feels like a really complex testing setup. Is there a simpler way? I'm wondering if there's a structural change we could make to simplify here.

Can I fixed these tests as a follow up?

Hweinstock · 2026-08-19T20:44:48Z

+ let parsed: unknown;
+ try {
+ parsed = JSON.parse(trimmed);
+ } catch {


should we wire the cause here?

Hweinstock · 2026-08-19T20:47:11Z

+import type { DatasetSchemaType } from "@aws-sdk/client-bedrock-agentcore-control";
+import type { InlineGroundTruth } from "@aws-sdk/client-bedrock-agentcore";
+
+// A record, not a bare string, so a future tool-branching type can widen it by a field.


i feel like the code explains this comment.

Hweinstock

only blocking question is why we do an explicit wait, rest could be polished later/in a follow-up. Is there a way we can be sure the spans we need exist before evaluating or is there a reason we don't have to be?

Hweinstock · 2026-08-24T14:39:15Z

+ throw new InputValidationError("required option '--name <name>' not specified");
+
+ // Ctrl-C aborts the run (invokes, the ingestion wait, the dataset download).
+ const controller = new AbortController();


looks like it was merged, so we should be able to leverage that shared abstraction now.
#1986

Hweinstock · 2026-08-24T14:42:18Z

+import { coreOptsFromCtx } from "../../../utils";
+import { parseRuntimeInvokeHeaders } from "../../../runtime/invoke/request";
+
+// Composes invokeDataset (replay) → startBatchEvaluation (grade). Invoke flags mirror


is the comment describing the implementation of the handler? What value does it provide over the code?

Hweinstock · 2026-08-24T16:13:18Z

+ // Enrich with the example identity so the dropped-invoke reason is self-describing
+ // in firstError, instead of a bare transport message logged separately.
+ const cause = error instanceof Error ? error : new Error(String(error));
+ throw new Error(


to confirm my understanding, this error never hits telemetry because an individual example failing does not bubble up so this error is purely for the user message it creates?

Correct. An individual example failing is caught and dropped by runExamples rather than rethrown, so it never reaches telemetry — it only feeds the user-facing message on a total failure. That was intentional: a run can be hundreds of examples, and I didn't want per-example failures flooding our telemetry.

Hweinstock · 2026-08-24T16:13:49Z

+
+ // AgentCore emits spans ~30s-3min after invoke; grade too early and it reads an empty
+ // log group and fails every session. Disabled via SIMULATE_INGESTION_WAIT_MS=0 (tests).
+ const waitMs = Number(process.env.SIMULATE_INGESTION_WAIT_MS ?? 180_000);


how do we know that there aren't traces still coming in after 3 min mark?

Hweinstock · 2026-08-24T16:17:43Z

+ try {
+ headers.set("Authorization", `Bearer ${bearerToken}`);
+ } catch {
+ throw new InputValidationError("Invalid bearer token");


should we wire the cause up to these errors to get the stack trace? (same with 132, 155 below)

Hweinstock · 2026-08-24T16:18:43Z

+ logger
+ .child({
+ errorName:
+ error instanceof TypeError


is there a reason error instanceof error ? error.name : 'error' doesn't work here?

I'll look into this in the next PR.

Hweinstock · 2026-08-24T16:21:33Z

+};
+
+// InvokeDatasetResult reports the created sessions plus how many examples were invoked
+// vs dropped (a failed invoke is skipped, not fatal). firstError explains a total failure.


does firstError explain a total failure? I thought it stored the first failed example's error?

WE will remove it later. Yes lets do a follow up on this.

Hweinstock · 2026-08-24T16:27:49Z

@@ -0,0 +1,263 @@
+// Disables the post-invoke span-ingestion wait so the replay returns immediately.
+process.env.SIMULATE_INGESTION_WAIT_MS = "0";


would this be a good application of global config? this might help this behavior be more discoverable by other tests that might need the same.

So add to the bunfig.toml to preload a script that run this:
./global-mocks.ts
// Global mocksimport{mock}from"bun:test";// Mock environment variablesprocess.env.SIMULATE_INGESTION_WAIT_MS="0";
bunfig.toml
[test]preload=["./global-mocks.ts"]
ref
Yes I could do that in the follow up Pr.

we should be able to inject the global config down through core with the data we want.

ah that make sense

Hweinstock · 2026-08-24T16:32:38Z

+ return this.groundTruth();
+ }
+
+ // Emit every turn (carrying its prompt), not just those with an expectation: filtering


i'm not quite sure what this comment is saying?

Ill remove it.

Hweinstock · 2026-08-24T21:27:20Z

Is there a way we can be sure the spans we need exist before evaluating or is there a reason we don't have to be?

from chatting offline, sounds like the wait approach is shared by the eval time. Still unsure how they can be confident the evaluator results are not incomplete, but approving since its an existing solution.

…example classes

- move renderJsonTemplate out of shared src/io into core/eval/invokeDataset - invokeRuntime: raw TypeError/Error -> InputValidationError/RuntimeInvokeResponseError - wire error causes in template + dataset JSON parse - invokeDataset: enrich per-example invoke failure instead of log+rethrow - simulate: bubble Ctrl-C cancellation (telemetry) instead of quiet return; clarify --description help; TODO(#1986) shared abort helper - drop type-guaranteed 'no leak' test; keep AbortSignal wiring in composition test - trim stale/redundant comments (runtime.tsx, invokeRuntime DTO note, TurnResult)

…in payload-template Bug bash (real exploratory-account run) surfaced two defects in the shared invokeDataset path: 1. PredefinedExample blind-cast assertions/expected_trajectory as string[] with no load-time validation (unlike turns). A non-array value passed load, burned a live paid invoke, then threw a raw '.map is not a function' mislabeled 'failed to invoke'. Now validated in the constructor -> clean InputValidationError before any invoke. 2. A --payload-template with no {input} placeholder was silently accepted, wasting a full ~3-min replay on a constant payload. Now rejected up front in invokeDataset.

nborges-aws · 2026-08-24T22:40:49Z

+ const sessionId = randomUUID();
+ const ctx: RunContext = {
+ invokeOnce: async (payload) => {
+ const response = await invokeRuntime(


Comparing this to src/handlers/runtime/invoke/request.ts, and I notice that the code calls normalizeRuntimeInvokeRequest to do some validation before invoking. Do you think we need something similar here?

nborges-aws

LGTM. One comment with a question for you

jariy17 · 2026-08-18T22:40:03Z

What

Adds batch-evaluation simulate to replay a dataset against a runtime and grade the resulting sessions.

Each dataset example is a self-running object that owns its own invoker (RunContext). invokeDataset builds one invoker per session and passes it into the example's run(). The example performs the invocation; the machine supplies the invoker and collects the result.

The invoker seam

A dataset example knows what to send and how to turn the responses into ground truth, but not how to reach the runtime. Its RunContext owns that runtime-specific invocation:

// src/core/eval/invokeDataset/example/types.tsexporttypeTurnResult={text: string};exporttypeRunContext={invokeOnce(input: string): Promise<TurnResult>};exportinterfaceExample{readonlyexampleId: string;readonlyschemaType: DatasetSchemaType;// Replay this example against the runtime `ctx` reaches, return neutral ground truth.run(ctx: RunContext): Promise<InlineGroundTruth|undefined>;}

invokeDataset resolves the runtime once, then builds a distinct invoker for each client-generated session and passes it to that example:

constresults=awaitrunExamples(examples,async(example)=>{constctx: RunContext={invokeOnce: async(input)=>{constres=awaitinvokeRuntime(deps,{/* resolved runtime, session, and rendered payload */},options,signal,);return{text: /* drained response body */};},};returnexample.run(ctx);});

The example decides how many times to call invokeOnce and in what order. Its invoker decides how each call reaches the runtime for that session.

Layout

src/core/eval/invokeDataset/
├── load.ts DatasetLoader — pure JSONL parse → shape-classify → new
├── run.ts runExamples — bounded-concurrency pool, failure isolation
├── invokeDataset.test.ts end-to-end golden coverage of the whole path
└── example/
├── types.ts Example interface, RunContext (the invoker), TurnResult
├── predefined.ts PredefinedExample — replays scripted turns, builds ground truth
└── simulated.ts SimulatedExample — not shipped; throws at construction

Plus src/core/invokeRuntime.ts, which extracts runtime invocation from runtime.tsx for reuse by both RuntimeClient and invokeDataset, and src/handlers/eval/batch-evaluation/simulate/index.tsx, which composes invokeDataset with startBatchEvaluation.

Testing

bun run typecheck passes.
bun test: 1544 pass, 0 fail.
invokeDataset.test.ts drives the real EvalClient.invokeDataset over a fake AWS layer with golden fixtures. Its snapshot covers created sessions and inline ground truth across every supported variation while exercising the loader, example classes, concurrency pool, template rendering, and runtime invocation.
simulate.test.tsx snapshots the handler's wrapped sessionMetadata.

codecov-commenter · 2026-08-18T22:42:42Z

Codecov Report

❌ Patch coverage is 96.58444% with 18 lines in your changes missing coverage. Please review.
✅ Project coverage is 97.33%. Comparing base (768ef10) to head (27c810f).

Files with missing lines	Patch %	Lines
src/core/eval.tsx	85.71%	14 Missing ⚠️
src/core/eval/invokeDataset/template.ts	91.30%	2 Missing ⚠️
src/core/eval/invokeDataset/example/simulated.ts	92.30%	1 Missing ⚠️
src/core/invokeRuntime.ts	99.35%	1 Missing ⚠️

Additional details and impacted files

@@ Coverage Diff @@## refactor #2032 +/- ##
============================================
- Coverage 97.36% 97.33% -0.04% 
============================================
Files 410 417 +7 Lines 24796 25181 +385 ============================================
+ Hits 24142 24509 +367 - Misses 654 672 +18

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Hweinstock

didn't get to all the examples work yet, but had a few small comments and a question on how we can simplify testing, because it feels pretty gnarly rn.

Hweinstock · 2026-08-19T19:55:13Z

@@ -0,0 +1,33 @@
+import { InputValidationError } from "../errors";


should we keep this inside evals until there is an opportunity to re-use it? I feel like the io directory should be reserved for shared abstractions.

Yeah, I was thinking the invoke handler could use this but lets leave it in the simulate handler for now.

Hweinstock · 2026-08-19T19:55:23Z

+ try {
+ parsed = JSON.parse(template);
+ } catch {
+ throw new InputValidationError(`--${flagName} must be valid JSON`);


should we wire the cause here?

Hweinstock · 2026-08-19T20:02:10Z

+ qualifier?: string;
+ payloadTemplate: string; // e.g. {"prompt":"{input}"} — {input} is the example's turn input
+ headers?: [string, string][];
+ bearerToken?: string;


q: is there a reason bearerToken is treated different from other headers?

It's the discriminator that selects the auth path, not an application header. Its presence routes the invoke to CUSTOM_JWT path (src/core/invokeRuntime.ts:73)

Hweinstock · 2026-08-19T20:07:33Z

+ flag("dataset-version", "dataset version (with a dataset id)", z.string().optional()),
+ flag("evaluator", "evaluator id(s) to apply", z.array(z.string()).optional()),
+ flag("name", "batch evaluation name (unique in the account)", z.string().optional()),
+ flag("description", "optional description", z.string().optional()),


what exactly is this describing? the simulation itself?

This should be batch-evaluation description. I'll rename this batch-eval-desc, same with name too.

Hweinstock · 2026-08-19T20:08:58Z

+ throw new InputValidationError("required option '--name <name>' not specified");
+
+ // Ctrl-C aborts the run (invokes, the ingestion wait, the dataset download).
+ const controller = new AbortController();


ahhh, theres a shared abstraction for this, but looks like it hasn't been merged yet :(
#1986
maybe we can swap it over as a follow-up once its merged.

looks like it was merged, so we should be able to leverage that shared abstraction now.
#1986

Hweinstock · 2026-08-19T20:33:35Z

+ const groundTruth = await example.run(ctx);
+ return { exampleId: example.exampleId, sessionId, groundTruth };
+ } catch (error) {
+ this.logger.debug(


instead of logging and rethrowing, is there a way to enrich the error thrown to avoid noise?

Hweinstock · 2026-08-19T20:34:33Z

+
+ // AgentCore emits spans ~30s-3min after invoke; grade too early and it reads an empty
+ // log group and fails every session. Disabled via SIMULATE_INGESTION_WAIT_MS=0 (tests).
+ const waitMs = Number(process.env.SIMULATE_INGESTION_WAIT_MS ?? 180_000);


is there anything we can poll on instead of a static wait time?

I didn't poll right away because some traces might still be arriving, and I didn't want us to end up with incomplete session data.

how do we know that there aren't traces still coming in after 3 min mark?

Hweinstock · 2026-08-19T20:44:21Z

+
+// A fake AWS layer: control resolves the runtime, data answers each invoke. Records every
+// payload it was asked to send, and per `opts` can fail or delay specific invokes.
+function fakeClients(opts: { fail?: (payload: string) => boolean; delayMs?: number } = {}): {


this feels like a really complex testing setup. Is there a simpler way? I'm wondering if there's a structural change we could make to simplify here.

Can I fixed these tests as a follow up?

Hweinstock · 2026-08-19T20:44:48Z

+ let parsed: unknown;
+ try {
+ parsed = JSON.parse(trimmed);
+ } catch {


should we wire the cause here?

Hweinstock · 2026-08-19T20:47:11Z

+import type { DatasetSchemaType } from "@aws-sdk/client-bedrock-agentcore-control";
+import type { InlineGroundTruth } from "@aws-sdk/client-bedrock-agentcore";
+
+// A record, not a bare string, so a future tool-branching type can widen it by a field.


i feel like the code explains this comment.

Hweinstock

only blocking question is why we do an explicit wait, rest could be polished later/in a follow-up. Is there a way we can be sure the spans we need exist before evaluating or is there a reason we don't have to be?

Hweinstock · 2026-08-24T14:39:15Z

+ throw new InputValidationError("required option '--name <name>' not specified");
+
+ // Ctrl-C aborts the run (invokes, the ingestion wait, the dataset download).
+ const controller = new AbortController();


looks like it was merged, so we should be able to leverage that shared abstraction now.
#1986

Hweinstock · 2026-08-24T14:42:18Z

+import { coreOptsFromCtx } from "../../../utils";
+import { parseRuntimeInvokeHeaders } from "../../../runtime/invoke/request";
+
+// Composes invokeDataset (replay) → startBatchEvaluation (grade). Invoke flags mirror


is the comment describing the implementation of the handler? What value does it provide over the code?

Hweinstock · 2026-08-24T16:13:18Z

+ // Enrich with the example identity so the dropped-invoke reason is self-describing
+ // in firstError, instead of a bare transport message logged separately.
+ const cause = error instanceof Error ? error : new Error(String(error));
+ throw new Error(


to confirm my understanding, this error never hits telemetry because an individual example failing does not bubble up so this error is purely for the user message it creates?

Correct. An individual example failing is caught and dropped by runExamples rather than rethrown, so it never reaches telemetry — it only feeds the user-facing message on a total failure. That was intentional: a run can be hundreds of examples, and I didn't want per-example failures flooding our telemetry.

Hweinstock · 2026-08-24T16:13:49Z

+
+ // AgentCore emits spans ~30s-3min after invoke; grade too early and it reads an empty
+ // log group and fails every session. Disabled via SIMULATE_INGESTION_WAIT_MS=0 (tests).
+ const waitMs = Number(process.env.SIMULATE_INGESTION_WAIT_MS ?? 180_000);


how do we know that there aren't traces still coming in after 3 min mark?

Hweinstock · 2026-08-24T16:17:43Z

+ try {
+ headers.set("Authorization", `Bearer ${bearerToken}`);
+ } catch {
+ throw new InputValidationError("Invalid bearer token");


should we wire the cause up to these errors to get the stack trace? (same with 132, 155 below)

Hweinstock · 2026-08-24T16:18:43Z

+ logger
+ .child({
+ errorName:
+ error instanceof TypeError


is there a reason error instanceof error ? error.name : 'error' doesn't work here?

I'll look into this in the next PR.

Hweinstock · 2026-08-24T16:21:33Z

+};
+
+// InvokeDatasetResult reports the created sessions plus how many examples were invoked
+// vs dropped (a failed invoke is skipped, not fatal). firstError explains a total failure.


does firstError explain a total failure? I thought it stored the first failed example's error?

WE will remove it later. Yes lets do a follow up on this.

Hweinstock · 2026-08-24T16:27:49Z

@@ -0,0 +1,263 @@
+// Disables the post-invoke span-ingestion wait so the replay returns immediately.
+process.env.SIMULATE_INGESTION_WAIT_MS = "0";


would this be a good application of global config? this might help this behavior be more discoverable by other tests that might need the same.

So add to the bunfig.toml to preload a script that run this:
./global-mocks.ts
// Global mocksimport{mock}from"bun:test";// Mock environment variablesprocess.env.SIMULATE_INGESTION_WAIT_MS="0";
bunfig.toml
[test]preload=["./global-mocks.ts"]
ref
Yes I could do that in the follow up Pr.

we should be able to inject the global config down through core with the data we want.

ah that make sense

Hweinstock · 2026-08-24T16:32:38Z

+ return this.groundTruth();
+ }
+
+ // Emit every turn (carrying its prompt), not just those with an expectation: filtering


i'm not quite sure what this comment is saying?

Ill remove it.

Hweinstock · 2026-08-24T21:27:20Z

Is there a way we can be sure the spans we need exist before evaluating or is there a reason we don't have to be?

from chatting offline, sounds like the wait approach is shared by the eval time. Still unsure how they can be confident the evaluator results are not incomplete, but approving since its an existing solution.

…example classes

- move renderJsonTemplate out of shared src/io into core/eval/invokeDataset - invokeRuntime: raw TypeError/Error -> InputValidationError/RuntimeInvokeResponseError - wire error causes in template + dataset JSON parse - invokeDataset: enrich per-example invoke failure instead of log+rethrow - simulate: bubble Ctrl-C cancellation (telemetry) instead of quiet return; clarify --description help; TODO(#1986) shared abort helper - drop type-guaranteed 'no leak' test; keep AbortSignal wiring in composition test - trim stale/redundant comments (runtime.tsx, invokeRuntime DTO note, TurnResult)

…in payload-template Bug bash (real exploratory-account run) surfaced two defects in the shared invokeDataset path: 1. PredefinedExample blind-cast assertions/expected_trajectory as string[] with no load-time validation (unlike turns). A non-array value passed load, burned a live paid invoke, then threw a raw '.map is not a function' mislabeled 'failed to invoke'. Now validated in the constructor -> clean InputValidationError before any invoke. 2. A --payload-template with no {input} placeholder was silently accepted, wasting a full ~3-min replay on a constant payload. Now rejected up front in invokeDataset.

nborges-aws · 2026-08-24T22:40:49Z

+ const sessionId = randomUUID();
+ const ctx: RunContext = {
+ invokeOnce: async (payload) => {
+ const response = await invokeRuntime(


Comparing this to src/handlers/runtime/invoke/request.ts, and I notice that the code calls normalizeRuntimeInvokeRequest to do some validation before invoking. Do you think we need something similar here?

nborges-aws

LGTM. One comment with a question for you

jariy17 mentioned this pull request Aug 18, 2026
feat(eval): add batch-evaluation simulate (dataset simulations) #2000
Closed

github-actionsBot added the agentcore-harness-reviewing AgentCore Harness review in progress label Aug 18, 2026

github-actionsBot removed the agentcore-harness-reviewing AgentCore Harness review in progress label Aug 18, 2026

jariy17 force-pushed the feat/eval-invoke-dataset-pr branch 6 times, most recently from 6a2915e to b747257Compare August 19, 2026 18:45

Hweinstock reviewed Aug 19, 2026
View reviewed changes

jariy17 requested a review from AlexanderRichey August 21, 2026 15:19

jariy17 mentioned this pull request Aug 22, 2026
feat(eval): ondemand simulate — replay a dataset, evaluate synchronously #2071
Merged

Hweinstock reviewed Aug 24, 2026
View reviewed changes

Hweinstock previously approved these changes Aug 24, 2026
View reviewed changes

jariy17 added 3 commits August 24, 2026 21:29

feat(eval): batch-evaluation simulate — invokeDataset + self-running …
75be9b4
…example classes

jariy17 dismissed Hweinstock’s stale review via 27c810fAugust 24, 2026 21:31

jariy17 force-pushed the feat/eval-invoke-dataset-pr branch from 708c074 to 27c810fCompare August 24, 2026 21:31

Hweinstock approved these changes Aug 24, 2026
View reviewed changes

nborges-aws reviewed Aug 24, 2026
View reviewed changes

nborges-aws approved these changes Aug 24, 2026
View reviewed changes

jariy17 merged commit 83dab8f into refactorAug 24, 2026
13 checks passed

jariy17 deleted the feat/eval-invoke-dataset-pr branch August 24, 2026 22:48

jariy17 mentioned this pull request Aug 25, 2026
feat(eval): batch simulate — --ingestion-wait-ms + per-example failures/sessions #2098
Merged