feat(eval): add batch-evaluation simulate (dataset simulations)

[jariy17]

Summary

simulate replays a dataset through a runtime — one client-side invoke per scenario — then submits a StartBatchEvaluation over the sessions it created.
The service has no dataset data source, so the CLI creates the sessions and the service grades them.

CLI command structure

agentcore eval batch-evaluation
├── evaluate (existing)
├── simulate ← this PR
├── get
└── list

Flags (mirror runtime invoke):

Flag	Req	Notes
--runtime-id	✅	runtime to invoke per scenario
--qualifier		endpoint qualifier (default DEFAULT)
--payload-template '{"prompt":"{input}"}'	✅	{input} = scenario input
--header		ordered app header (repeatable)
--bearer-token		CUSTOM_JWT bearer token
--user-id		runtime user id
--dataset <path|id>	✅	local JSONL path or dataset id
--dataset-version		with a dataset id
--evaluator	✅	evaluator id(s)
--name	✅	unique in the account
--description		optional
--kms-key-arn		encrypt eval data at rest

content-type / accept are hardcoded application/json.

How it works

1. Load dataset — local JSONL path, or fetch by dataset id.
2. Invoke agent per scenario — concurrent, one fresh client-generated session id per scenario. Multi-turn = sequential invokes on one session.
3. Wait ~180s for span ingestion (spans must land before grading reads them).
4. StartBatchEvaluation — cloudWatchLogs scoped to those sessionIds + evaluationMetadata.sessionMetadata.
5. Poll with agentcore eval batch-evaluation get.

Design decisions

• Method on EvalClient — simulate is a method, no new class.
• Extracted invokeRuntime free fns to src/core/invokeRuntime.ts — no sibling core.runtime.* calls, star graph preserved; RuntimeClient.invokeRuntime becomes a 1-line delegate.
• Handler owns the public contract only — SimulateInput / SimulateResult are public; private helpers use local types (no handler-interface leaks inward).
• Generalized renderJsonTemplate in src/io — JSON-safe {key} substitution, not eval-specific.
• Session id client-generated — client-owned per AWS docs (min 33 chars; randomUUID() = 36), needed for microVM stickiness + multi-turn threading; matches the public bedrock_agentcore SDK rather than reading a response header.
• Ctrl-C cancellable — AbortSignal threaded through handler → core → invoke/sleep; aborts in-flight invokes, the dataset download, and the ingestion wait.

Testing

Live-verified in account 685197708687 / us-west-2, agent asdf_MyAgent-3s5axvBC6Q, evaluator Builtin.Helpfulness:

• Single-turn job sim_live_1f9e-cbcc074b38 → COMPLETED, 2/2 sessions graded.
• Multi-turn job sim_mt_2755-7517683065 (2-turn "remember 7 → recall it") → COMPLETED, 2/2 graded, 2 sessions scoped.

Automated:

• bun test src/ → 1383 pass; typecheck + oxlint clean.
• Unit tests: src/io/template.test.ts, src/handlers/eval/batch-evaluation/simulate/simulate.test.tsx.

--help (captured via tui-harness):

Usage: agentcore eval batch-evaluation simulate [options]
replay a dataset against a runtime, then batch-evaluate the resulting sessions
Options:
--runtime-id <runtime-id> runtime id to invoke per scenario
--qualifier <qualifier> runtime endpoint qualifier (default DEFAULT)
--payload-template <payload-template> JSON payload template; {input} is the scenario input, e.g.
{"prompt":"{input}"}
--header <header...> an ordered application header (repeatable)
--bearer-token <bearer-token> CUSTOM_JWT bearer token (for JWT-auth runtimes)
--user-id <user-id> runtime user id
--dataset <dataset> dataset source: local JSONL path or a dataset id
--dataset-version <dataset-version> dataset version (with a dataset id)
--evaluator <evaluator...> evaluator id(s) to apply
--name <name> batch evaluation name (unique in the account)
--description <description> optional description
--kms-key-arn <kms-key-arn> KMS key to encrypt evaluation data at rest
-h, --help display help for command

Required-flag validation (captured via tui-harness, --name omitted):

$ agentcore eval batch-evaluation simulate --dataset /tmp/x.jsonl --runtime-id r --payload-template '{}' --evaluator E
Error: required option '--name <name>' not specified

[codecov-commenter]

Codecov Report

❌ Patch coverage is 74.42159% with 199 lines in your changes missing coverage. Please review.
✅ Project coverage is 96.12%. Comparing base (4d183dc) to head (80291f9).
⚠️ Report is 1 commits behind head on refactor.

Files with missing lines	Patch %	Lines
src/core/eval.tsx	65.23%	113 Missing ⚠️
src/core/eval/simulate.ts	4.70%	81 Missing ⚠️
src/handlers/eval/ondemand/evaluate/index.tsx	98.16%	2 Missing ⚠️
src/io/template.ts	91.30%	2 Missing ⚠️
src/core/invokeRuntime.ts	99.33%	1 Missing ⚠️

Additional details and impacted files

@@ Coverage Diff @@## refactor #2000 +/- ##
============================================
- Coverage 96.96% 96.12% -0.84% 
============================================
Files 364 370 +6 Lines 20758 21393 +635 ============================================
+ Hits 20127 20563 +436 - Misses 631 830 +199 

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[jariy17]

Superseded by #2032, which folds this replay path into the invokeDataset design (each example owns its invoker, passed into run()). Closing in favor of that.

[jariy17]

Summary

simulate replays a dataset through a runtime — one client-side invoke per scenario — then submits a StartBatchEvaluation over the sessions it created.
The service has no dataset data source, so the CLI creates the sessions and the service grades them.

CLI command structure

agentcore eval batch-evaluation
├── evaluate (existing)
├── simulate ← this PR
├── get
└── list

Flags (mirror runtime invoke):

Flag	Req	Notes
--runtime-id	✅	runtime to invoke per scenario
--qualifier		endpoint qualifier (default DEFAULT)
--payload-template '{"prompt":"{input}"}'	✅	{input} = scenario input
--header		ordered app header (repeatable)
--bearer-token		CUSTOM_JWT bearer token
--user-id		runtime user id
--dataset <path|id>	✅	local JSONL path or dataset id
--dataset-version		with a dataset id
--evaluator	✅	evaluator id(s)
--name	✅	unique in the account
--description		optional
--kms-key-arn		encrypt eval data at rest

content-type / accept are hardcoded application/json.

How it works

1. Load dataset — local JSONL path, or fetch by dataset id.
2. Invoke agent per scenario — concurrent, one fresh client-generated session id per scenario. Multi-turn = sequential invokes on one session.
3. Wait ~180s for span ingestion (spans must land before grading reads them).
4. StartBatchEvaluation — cloudWatchLogs scoped to those sessionIds + evaluationMetadata.sessionMetadata.
5. Poll with agentcore eval batch-evaluation get.

Design decisions

• Method on EvalClient — simulate is a method, no new class.
• Extracted invokeRuntime free fns to src/core/invokeRuntime.ts — no sibling core.runtime.* calls, star graph preserved; RuntimeClient.invokeRuntime becomes a 1-line delegate.
• Handler owns the public contract only — SimulateInput / SimulateResult are public; private helpers use local types (no handler-interface leaks inward).
• Generalized renderJsonTemplate in src/io — JSON-safe {key} substitution, not eval-specific.
• Session id client-generated — client-owned per AWS docs (min 33 chars; randomUUID() = 36), needed for microVM stickiness + multi-turn threading; matches the public bedrock_agentcore SDK rather than reading a response header.
• Ctrl-C cancellable — AbortSignal threaded through handler → core → invoke/sleep; aborts in-flight invokes, the dataset download, and the ingestion wait.

Testing

Live-verified in account 685197708687 / us-west-2, agent asdf_MyAgent-3s5axvBC6Q, evaluator Builtin.Helpfulness:

• Single-turn job sim_live_1f9e-cbcc074b38 → COMPLETED, 2/2 sessions graded.
• Multi-turn job sim_mt_2755-7517683065 (2-turn "remember 7 → recall it") → COMPLETED, 2/2 graded, 2 sessions scoped.

Automated:

• bun test src/ → 1383 pass; typecheck + oxlint clean.
• Unit tests: src/io/template.test.ts, src/handlers/eval/batch-evaluation/simulate/simulate.test.tsx.

--help (captured via tui-harness):

Usage: agentcore eval batch-evaluation simulate [options]
replay a dataset against a runtime, then batch-evaluate the resulting sessions
Options:
--runtime-id <runtime-id> runtime id to invoke per scenario
--qualifier <qualifier> runtime endpoint qualifier (default DEFAULT)
--payload-template <payload-template> JSON payload template; {input} is the scenario input, e.g.
{"prompt":"{input}"}
--header <header...> an ordered application header (repeatable)
--bearer-token <bearer-token> CUSTOM_JWT bearer token (for JWT-auth runtimes)
--user-id <user-id> runtime user id
--dataset <dataset> dataset source: local JSONL path or a dataset id
--dataset-version <dataset-version> dataset version (with a dataset id)
--evaluator <evaluator...> evaluator id(s) to apply
--name <name> batch evaluation name (unique in the account)
--description <description> optional description
--kms-key-arn <kms-key-arn> KMS key to encrypt evaluation data at rest
-h, --help display help for command

Required-flag validation (captured via tui-harness, --name omitted):

$ agentcore eval batch-evaluation simulate --dataset /tmp/x.jsonl --runtime-id r --payload-template '{}' --evaluator E
Error: required option '--name <name>' not specified

[codecov-commenter]

Codecov Report

❌ Patch coverage is 74.42159% with 199 lines in your changes missing coverage. Please review.
✅ Project coverage is 96.12%. Comparing base (4d183dc) to head (80291f9).
⚠️ Report is 1 commits behind head on refactor.

Files with missing lines	Patch %	Lines
src/core/eval.tsx	65.23%	113 Missing ⚠️
src/core/eval/simulate.ts	4.70%	81 Missing ⚠️
src/handlers/eval/ondemand/evaluate/index.tsx	98.16%	2 Missing ⚠️
src/io/template.ts	91.30%	2 Missing ⚠️
src/core/invokeRuntime.ts	99.33%	1 Missing ⚠️

Additional details and impacted files

@@ Coverage Diff @@## refactor #2000 +/- ##
============================================
- Coverage 96.96% 96.12% -0.84% 
============================================
Files 364 370 +6 Lines 20758 21393 +635 ============================================
+ Hits 20127 20563 +436 - Misses 631 830 +199 

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[jariy17]

Superseded by #2032, which folds this replay path into the invokeDataset design (each example owns its invoker, passed into run()). Closing in favor of that.

[jariy17]

Summary

simulate replays a dataset through a runtime — one client-side invoke per scenario — then submits a StartBatchEvaluation over the sessions it created.
The service has no dataset data source, so the CLI creates the sessions and the service grades them.

CLI command structure

agentcore eval batch-evaluation
├── evaluate (existing)
├── simulate ← this PR
├── get
└── list

Flags (mirror runtime invoke):

Flag	Req	Notes
--runtime-id	✅	runtime to invoke per scenario
--qualifier		endpoint qualifier (default DEFAULT)
--payload-template '{"prompt":"{input}"}'	✅	{input} = scenario input
--header		ordered app header (repeatable)
--bearer-token		CUSTOM_JWT bearer token
--user-id		runtime user id
--dataset <path|id>	✅	local JSONL path or dataset id
--dataset-version		with a dataset id
--evaluator	✅	evaluator id(s)
--name	✅	unique in the account
--description		optional
--kms-key-arn		encrypt eval data at rest

content-type / accept are hardcoded application/json.

How it works

1. Load dataset — local JSONL path, or fetch by dataset id.
2. Invoke agent per scenario — concurrent, one fresh client-generated session id per scenario. Multi-turn = sequential invokes on one session.
3. Wait ~180s for span ingestion (spans must land before grading reads them).
4. StartBatchEvaluation — cloudWatchLogs scoped to those sessionIds + evaluationMetadata.sessionMetadata.
5. Poll with agentcore eval batch-evaluation get.

Design decisions

• Method on EvalClient — simulate is a method, no new class.
• Extracted invokeRuntime free fns to src/core/invokeRuntime.ts — no sibling core.runtime.* calls, star graph preserved; RuntimeClient.invokeRuntime becomes a 1-line delegate.
• Handler owns the public contract only — SimulateInput / SimulateResult are public; private helpers use local types (no handler-interface leaks inward).
• Generalized renderJsonTemplate in src/io — JSON-safe {key} substitution, not eval-specific.
• Session id client-generated — client-owned per AWS docs (min 33 chars; randomUUID() = 36), needed for microVM stickiness + multi-turn threading; matches the public bedrock_agentcore SDK rather than reading a response header.
• Ctrl-C cancellable — AbortSignal threaded through handler → core → invoke/sleep; aborts in-flight invokes, the dataset download, and the ingestion wait.

Testing

Live-verified in account 685197708687 / us-west-2, agent asdf_MyAgent-3s5axvBC6Q, evaluator Builtin.Helpfulness:

• Single-turn job sim_live_1f9e-cbcc074b38 → COMPLETED, 2/2 sessions graded.
• Multi-turn job sim_mt_2755-7517683065 (2-turn "remember 7 → recall it") → COMPLETED, 2/2 graded, 2 sessions scoped.

Automated:

• bun test src/ → 1383 pass; typecheck + oxlint clean.
• Unit tests: src/io/template.test.ts, src/handlers/eval/batch-evaluation/simulate/simulate.test.tsx.

--help (captured via tui-harness):

Usage: agentcore eval batch-evaluation simulate [options]
replay a dataset against a runtime, then batch-evaluate the resulting sessions
Options:
--runtime-id <runtime-id> runtime id to invoke per scenario
--qualifier <qualifier> runtime endpoint qualifier (default DEFAULT)
--payload-template <payload-template> JSON payload template; {input} is the scenario input, e.g.
{"prompt":"{input}"}
--header <header...> an ordered application header (repeatable)
--bearer-token <bearer-token> CUSTOM_JWT bearer token (for JWT-auth runtimes)
--user-id <user-id> runtime user id
--dataset <dataset> dataset source: local JSONL path or a dataset id
--dataset-version <dataset-version> dataset version (with a dataset id)
--evaluator <evaluator...> evaluator id(s) to apply
--name <name> batch evaluation name (unique in the account)
--description <description> optional description
--kms-key-arn <kms-key-arn> KMS key to encrypt evaluation data at rest
-h, --help display help for command

Required-flag validation (captured via tui-harness, --name omitted):

$ agentcore eval batch-evaluation simulate --dataset /tmp/x.jsonl --runtime-id r --payload-template '{}' --evaluator E
Error: required option '--name <name>' not specified

[codecov-commenter]

Codecov Report

❌ Patch coverage is 74.42159% with 199 lines in your changes missing coverage. Please review.
✅ Project coverage is 96.12%. Comparing base (4d183dc) to head (80291f9).
⚠️ Report is 1 commits behind head on refactor.

Files with missing lines	Patch %	Lines
src/core/eval.tsx	65.23%	113 Missing ⚠️
src/core/eval/simulate.ts	4.70%	81 Missing ⚠️
src/handlers/eval/ondemand/evaluate/index.tsx	98.16%	2 Missing ⚠️
src/io/template.ts	91.30%	2 Missing ⚠️
src/core/invokeRuntime.ts	99.33%	1 Missing ⚠️

Additional details and impacted files

@@ Coverage Diff @@## refactor #2000 +/- ##
============================================
- Coverage 96.96% 96.12% -0.84% 
============================================
Files 364 370 +6 Lines 20758 21393 +635 ============================================
+ Hits 20127 20563 +436 - Misses 631 830 +199 

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[jariy17]

Superseded by #2032, which folds this replay path into the invokeDataset design (each example owns its invoker, passed into run()). Closing in favor of that.

[jariy17]

Summary

simulate replays a dataset through a runtime — one client-side invoke per scenario — then submits a StartBatchEvaluation over the sessions it created.
The service has no dataset data source, so the CLI creates the sessions and the service grades them.

CLI command structure

agentcore eval batch-evaluation
├── evaluate (existing)
├── simulate ← this PR
├── get
└── list

Flags (mirror runtime invoke):

Flag	Req	Notes
--runtime-id	✅	runtime to invoke per scenario
--qualifier		endpoint qualifier (default DEFAULT)
--payload-template '{"prompt":"{input}"}'	✅	{input} = scenario input
--header		ordered app header (repeatable)
--bearer-token		CUSTOM_JWT bearer token
--user-id		runtime user id
--dataset <path|id>	✅	local JSONL path or dataset id
--dataset-version		with a dataset id
--evaluator	✅	evaluator id(s)
--name	✅	unique in the account
--description		optional
--kms-key-arn		encrypt eval data at rest

content-type / accept are hardcoded application/json.

How it works

1. Load dataset — local JSONL path, or fetch by dataset id.
2. Invoke agent per scenario — concurrent, one fresh client-generated session id per scenario. Multi-turn = sequential invokes on one session.
3. Wait ~180s for span ingestion (spans must land before grading reads them).
4. StartBatchEvaluation — cloudWatchLogs scoped to those sessionIds + evaluationMetadata.sessionMetadata.
5. Poll with agentcore eval batch-evaluation get.

Design decisions

• Method on EvalClient — simulate is a method, no new class.
• Extracted invokeRuntime free fns to src/core/invokeRuntime.ts — no sibling core.runtime.* calls, star graph preserved; RuntimeClient.invokeRuntime becomes a 1-line delegate.
• Handler owns the public contract only — SimulateInput / SimulateResult are public; private helpers use local types (no handler-interface leaks inward).
• Generalized renderJsonTemplate in src/io — JSON-safe {key} substitution, not eval-specific.
• Session id client-generated — client-owned per AWS docs (min 33 chars; randomUUID() = 36), needed for microVM stickiness + multi-turn threading; matches the public bedrock_agentcore SDK rather than reading a response header.
• Ctrl-C cancellable — AbortSignal threaded through handler → core → invoke/sleep; aborts in-flight invokes, the dataset download, and the ingestion wait.

Testing

Live-verified in account 685197708687 / us-west-2, agent asdf_MyAgent-3s5axvBC6Q, evaluator Builtin.Helpfulness:

• Single-turn job sim_live_1f9e-cbcc074b38 → COMPLETED, 2/2 sessions graded.
• Multi-turn job sim_mt_2755-7517683065 (2-turn "remember 7 → recall it") → COMPLETED, 2/2 graded, 2 sessions scoped.

Automated:

• bun test src/ → 1383 pass; typecheck + oxlint clean.
• Unit tests: src/io/template.test.ts, src/handlers/eval/batch-evaluation/simulate/simulate.test.tsx.

--help (captured via tui-harness):

Usage: agentcore eval batch-evaluation simulate [options]
replay a dataset against a runtime, then batch-evaluate the resulting sessions
Options:
--runtime-id <runtime-id> runtime id to invoke per scenario
--qualifier <qualifier> runtime endpoint qualifier (default DEFAULT)
--payload-template <payload-template> JSON payload template; {input} is the scenario input, e.g.
{"prompt":"{input}"}
--header <header...> an ordered application header (repeatable)
--bearer-token <bearer-token> CUSTOM_JWT bearer token (for JWT-auth runtimes)
--user-id <user-id> runtime user id
--dataset <dataset> dataset source: local JSONL path or a dataset id
--dataset-version <dataset-version> dataset version (with a dataset id)
--evaluator <evaluator...> evaluator id(s) to apply
--name <name> batch evaluation name (unique in the account)
--description <description> optional description
--kms-key-arn <kms-key-arn> KMS key to encrypt evaluation data at rest
-h, --help display help for command

Required-flag validation (captured via tui-harness, --name omitted):

$ agentcore eval batch-evaluation simulate --dataset /tmp/x.jsonl --runtime-id r --payload-template '{}' --evaluator E
Error: required option '--name <name>' not specified

[codecov-commenter]

Codecov Report

❌ Patch coverage is 74.42159% with 199 lines in your changes missing coverage. Please review.
✅ Project coverage is 96.12%. Comparing base (4d183dc) to head (80291f9).
⚠️ Report is 1 commits behind head on refactor.

Files with missing lines	Patch %	Lines
src/core/eval.tsx	65.23%	113 Missing ⚠️
src/core/eval/simulate.ts	4.70%	81 Missing ⚠️
src/handlers/eval/ondemand/evaluate/index.tsx	98.16%	2 Missing ⚠️
src/io/template.ts	91.30%	2 Missing ⚠️
src/core/invokeRuntime.ts	99.33%	1 Missing ⚠️

Additional details and impacted files

@@ Coverage Diff @@## refactor #2000 +/- ##
============================================
- Coverage 96.96% 96.12% -0.84% 
============================================
Files 364 370 +6 Lines 20758 21393 +635 ============================================
+ Hits 20127 20563 +436 - Misses 631 830 +199 

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[jariy17]

Superseded by #2032, which folds this replay path into the invokeDataset design (each example owns its invoker, passed into run()). Closing in favor of that.

[jariy17]

Summary

simulate replays a dataset through a runtime — one client-side invoke per scenario — then submits a StartBatchEvaluation over the sessions it created.
The service has no dataset data source, so the CLI creates the sessions and the service grades them.

CLI command structure

agentcore eval batch-evaluation
├── evaluate (existing)
├── simulate ← this PR
├── get
└── list

Flags (mirror runtime invoke):

Flag	Req	Notes
--runtime-id	✅	runtime to invoke per scenario
--qualifier		endpoint qualifier (default DEFAULT)
--payload-template '{"prompt":"{input}"}'	✅	{input} = scenario input
--header		ordered app header (repeatable)
--bearer-token		CUSTOM_JWT bearer token
--user-id		runtime user id
--dataset <path|id>	✅	local JSONL path or dataset id
--dataset-version		with a dataset id
--evaluator	✅	evaluator id(s)
--name	✅	unique in the account
--description		optional
--kms-key-arn		encrypt eval data at rest

content-type / accept are hardcoded application/json.

How it works

1. Load dataset — local JSONL path, or fetch by dataset id.
2. Invoke agent per scenario — concurrent, one fresh client-generated session id per scenario. Multi-turn = sequential invokes on one session.
3. Wait ~180s for span ingestion (spans must land before grading reads them).
4. StartBatchEvaluation — cloudWatchLogs scoped to those sessionIds + evaluationMetadata.sessionMetadata.
5. Poll with agentcore eval batch-evaluation get.

Design decisions

• Method on EvalClient — simulate is a method, no new class.
• Extracted invokeRuntime free fns to src/core/invokeRuntime.ts — no sibling core.runtime.* calls, star graph preserved; RuntimeClient.invokeRuntime becomes a 1-line delegate.
• Handler owns the public contract only — SimulateInput / SimulateResult are public; private helpers use local types (no handler-interface leaks inward).
• Generalized renderJsonTemplate in src/io — JSON-safe {key} substitution, not eval-specific.
• Session id client-generated — client-owned per AWS docs (min 33 chars; randomUUID() = 36), needed for microVM stickiness + multi-turn threading; matches the public bedrock_agentcore SDK rather than reading a response header.
• Ctrl-C cancellable — AbortSignal threaded through handler → core → invoke/sleep; aborts in-flight invokes, the dataset download, and the ingestion wait.

Testing

Live-verified in account 685197708687 / us-west-2, agent asdf_MyAgent-3s5axvBC6Q, evaluator Builtin.Helpfulness:

• Single-turn job sim_live_1f9e-cbcc074b38 → COMPLETED, 2/2 sessions graded.
• Multi-turn job sim_mt_2755-7517683065 (2-turn "remember 7 → recall it") → COMPLETED, 2/2 graded, 2 sessions scoped.

Automated:

• bun test src/ → 1383 pass; typecheck + oxlint clean.
• Unit tests: src/io/template.test.ts, src/handlers/eval/batch-evaluation/simulate/simulate.test.tsx.

--help (captured via tui-harness):

Usage: agentcore eval batch-evaluation simulate [options]
replay a dataset against a runtime, then batch-evaluate the resulting sessions
Options:
--runtime-id <runtime-id> runtime id to invoke per scenario
--qualifier <qualifier> runtime endpoint qualifier (default DEFAULT)
--payload-template <payload-template> JSON payload template; {input} is the scenario input, e.g.
{"prompt":"{input}"}
--header <header...> an ordered application header (repeatable)
--bearer-token <bearer-token> CUSTOM_JWT bearer token (for JWT-auth runtimes)
--user-id <user-id> runtime user id
--dataset <dataset> dataset source: local JSONL path or a dataset id
--dataset-version <dataset-version> dataset version (with a dataset id)
--evaluator <evaluator...> evaluator id(s) to apply
--name <name> batch evaluation name (unique in the account)
--description <description> optional description
--kms-key-arn <kms-key-arn> KMS key to encrypt evaluation data at rest
-h, --help display help for command

Required-flag validation (captured via tui-harness, --name omitted):

$ agentcore eval batch-evaluation simulate --dataset /tmp/x.jsonl --runtime-id r --payload-template '{}' --evaluator E
Error: required option '--name <name>' not specified

[codecov-commenter]

Codecov Report

❌ Patch coverage is 74.42159% with 199 lines in your changes missing coverage. Please review.
✅ Project coverage is 96.12%. Comparing base (4d183dc) to head (80291f9).
⚠️ Report is 1 commits behind head on refactor.

Files with missing lines	Patch %	Lines
src/core/eval.tsx	65.23%	113 Missing ⚠️
src/core/eval/simulate.ts	4.70%	81 Missing ⚠️
src/handlers/eval/ondemand/evaluate/index.tsx	98.16%	2 Missing ⚠️
src/io/template.ts	91.30%	2 Missing ⚠️
src/core/invokeRuntime.ts	99.33%	1 Missing ⚠️

Additional details and impacted files

@@ Coverage Diff @@## refactor #2000 +/- ##
============================================
- Coverage 96.96% 96.12% -0.84% 
============================================
Files 364 370 +6 Lines 20758 21393 +635 ============================================
+ Hits 20127 20563 +436 - Misses 631 830 +199 

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[jariy17]

Superseded by #2032, which folds this replay path into the invokeDataset design (each example owns its invoker, passed into run()). Closing in favor of that.

[jariy17]

Summary

simulate replays a dataset through a runtime — one client-side invoke per scenario — then submits a StartBatchEvaluation over the sessions it created.
The service has no dataset data source, so the CLI creates the sessions and the service grades them.

CLI command structure

agentcore eval batch-evaluation
├── evaluate (existing)
├── simulate ← this PR
├── get
└── list

Flags (mirror runtime invoke):

Flag	Req	Notes
--runtime-id	✅	runtime to invoke per scenario
--qualifier		endpoint qualifier (default DEFAULT)
--payload-template '{"prompt":"{input}"}'	✅	{input} = scenario input
--header		ordered app header (repeatable)
--bearer-token		CUSTOM_JWT bearer token
--user-id		runtime user id
--dataset <path|id>	✅	local JSONL path or dataset id
--dataset-version		with a dataset id
--evaluator	✅	evaluator id(s)
--name	✅	unique in the account
--description		optional
--kms-key-arn		encrypt eval data at rest

content-type / accept are hardcoded application/json.

How it works

1. Load dataset — local JSONL path, or fetch by dataset id.
2. Invoke agent per scenario — concurrent, one fresh client-generated session id per scenario. Multi-turn = sequential invokes on one session.
3. Wait ~180s for span ingestion (spans must land before grading reads them).
4. StartBatchEvaluation — cloudWatchLogs scoped to those sessionIds + evaluationMetadata.sessionMetadata.
5. Poll with agentcore eval batch-evaluation get.

Design decisions

• Method on EvalClient — simulate is a method, no new class.
• Extracted invokeRuntime free fns to src/core/invokeRuntime.ts — no sibling core.runtime.* calls, star graph preserved; RuntimeClient.invokeRuntime becomes a 1-line delegate.
• Handler owns the public contract only — SimulateInput / SimulateResult are public; private helpers use local types (no handler-interface leaks inward).
• Generalized renderJsonTemplate in src/io — JSON-safe {key} substitution, not eval-specific.
• Session id client-generated — client-owned per AWS docs (min 33 chars; randomUUID() = 36), needed for microVM stickiness + multi-turn threading; matches the public bedrock_agentcore SDK rather than reading a response header.
• Ctrl-C cancellable — AbortSignal threaded through handler → core → invoke/sleep; aborts in-flight invokes, the dataset download, and the ingestion wait.

Testing

Live-verified in account 685197708687 / us-west-2, agent asdf_MyAgent-3s5axvBC6Q, evaluator Builtin.Helpfulness:

• Single-turn job sim_live_1f9e-cbcc074b38 → COMPLETED, 2/2 sessions graded.
• Multi-turn job sim_mt_2755-7517683065 (2-turn "remember 7 → recall it") → COMPLETED, 2/2 graded, 2 sessions scoped.

Automated:

• bun test src/ → 1383 pass; typecheck + oxlint clean.
• Unit tests: src/io/template.test.ts, src/handlers/eval/batch-evaluation/simulate/simulate.test.tsx.

--help (captured via tui-harness):

Usage: agentcore eval batch-evaluation simulate [options]
replay a dataset against a runtime, then batch-evaluate the resulting sessions
Options:
--runtime-id <runtime-id> runtime id to invoke per scenario
--qualifier <qualifier> runtime endpoint qualifier (default DEFAULT)
--payload-template <payload-template> JSON payload template; {input} is the scenario input, e.g.
{"prompt":"{input}"}
--header <header...> an ordered application header (repeatable)
--bearer-token <bearer-token> CUSTOM_JWT bearer token (for JWT-auth runtimes)
--user-id <user-id> runtime user id
--dataset <dataset> dataset source: local JSONL path or a dataset id
--dataset-version <dataset-version> dataset version (with a dataset id)
--evaluator <evaluator...> evaluator id(s) to apply
--name <name> batch evaluation name (unique in the account)
--description <description> optional description
--kms-key-arn <kms-key-arn> KMS key to encrypt evaluation data at rest
-h, --help display help for command

Required-flag validation (captured via tui-harness, --name omitted):

$ agentcore eval batch-evaluation simulate --dataset /tmp/x.jsonl --runtime-id r --payload-template '{}' --evaluator E
Error: required option '--name <name>' not specified

[codecov-commenter]

Codecov Report

❌ Patch coverage is 74.42159% with 199 lines in your changes missing coverage. Please review.
✅ Project coverage is 96.12%. Comparing base (4d183dc) to head (80291f9).
⚠️ Report is 1 commits behind head on refactor.

Files with missing lines	Patch %	Lines
src/core/eval.tsx	65.23%	113 Missing ⚠️
src/core/eval/simulate.ts	4.70%	81 Missing ⚠️
src/handlers/eval/ondemand/evaluate/index.tsx	98.16%	2 Missing ⚠️
src/io/template.ts	91.30%	2 Missing ⚠️
src/core/invokeRuntime.ts	99.33%	1 Missing ⚠️

Additional details and impacted files

@@ Coverage Diff @@## refactor #2000 +/- ##
============================================
- Coverage 96.96% 96.12% -0.84% 
============================================
Files 364 370 +6 Lines 20758 21393 +635 ============================================
+ Hits 20127 20563 +436 - Misses 631 830 +199 

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[jariy17]

Superseded by #2032, which folds this replay path into the invokeDataset design (each example owns its invoker, passed into run()). Closing in favor of that.

[jariy17]

Summary

simulate replays a dataset through a runtime — one client-side invoke per scenario — then submits a StartBatchEvaluation over the sessions it created.
The service has no dataset data source, so the CLI creates the sessions and the service grades them.

CLI command structure

agentcore eval batch-evaluation
├── evaluate (existing)
├── simulate ← this PR
├── get
└── list

Flags (mirror runtime invoke):

Flag	Req	Notes
--runtime-id	✅	runtime to invoke per scenario
--qualifier		endpoint qualifier (default DEFAULT)
--payload-template '{"prompt":"{input}"}'	✅	{input} = scenario input
--header		ordered app header (repeatable)
--bearer-token		CUSTOM_JWT bearer token
--user-id		runtime user id
--dataset <path|id>	✅	local JSONL path or dataset id
--dataset-version		with a dataset id
--evaluator	✅	evaluator id(s)
--name	✅	unique in the account
--description		optional
--kms-key-arn		encrypt eval data at rest

content-type / accept are hardcoded application/json.

How it works

1. Load dataset — local JSONL path, or fetch by dataset id.
2. Invoke agent per scenario — concurrent, one fresh client-generated session id per scenario. Multi-turn = sequential invokes on one session.
3. Wait ~180s for span ingestion (spans must land before grading reads them).
4. StartBatchEvaluation — cloudWatchLogs scoped to those sessionIds + evaluationMetadata.sessionMetadata.
5. Poll with agentcore eval batch-evaluation get.

Design decisions

• Method on EvalClient — simulate is a method, no new class.
• Extracted invokeRuntime free fns to src/core/invokeRuntime.ts — no sibling core.runtime.* calls, star graph preserved; RuntimeClient.invokeRuntime becomes a 1-line delegate.
• Handler owns the public contract only — SimulateInput / SimulateResult are public; private helpers use local types (no handler-interface leaks inward).
• Generalized renderJsonTemplate in src/io — JSON-safe {key} substitution, not eval-specific.
• Session id client-generated — client-owned per AWS docs (min 33 chars; randomUUID() = 36), needed for microVM stickiness + multi-turn threading; matches the public bedrock_agentcore SDK rather than reading a response header.
• Ctrl-C cancellable — AbortSignal threaded through handler → core → invoke/sleep; aborts in-flight invokes, the dataset download, and the ingestion wait.

Testing

Live-verified in account 685197708687 / us-west-2, agent asdf_MyAgent-3s5axvBC6Q, evaluator Builtin.Helpfulness:

• Single-turn job sim_live_1f9e-cbcc074b38 → COMPLETED, 2/2 sessions graded.
• Multi-turn job sim_mt_2755-7517683065 (2-turn "remember 7 → recall it") → COMPLETED, 2/2 graded, 2 sessions scoped.

Automated:

• bun test src/ → 1383 pass; typecheck + oxlint clean.
• Unit tests: src/io/template.test.ts, src/handlers/eval/batch-evaluation/simulate/simulate.test.tsx.

--help (captured via tui-harness):

Usage: agentcore eval batch-evaluation simulate [options]
replay a dataset against a runtime, then batch-evaluate the resulting sessions
Options:
--runtime-id <runtime-id> runtime id to invoke per scenario
--qualifier <qualifier> runtime endpoint qualifier (default DEFAULT)
--payload-template <payload-template> JSON payload template; {input} is the scenario input, e.g.
{"prompt":"{input}"}
--header <header...> an ordered application header (repeatable)
--bearer-token <bearer-token> CUSTOM_JWT bearer token (for JWT-auth runtimes)
--user-id <user-id> runtime user id
--dataset <dataset> dataset source: local JSONL path or a dataset id
--dataset-version <dataset-version> dataset version (with a dataset id)
--evaluator <evaluator...> evaluator id(s) to apply
--name <name> batch evaluation name (unique in the account)
--description <description> optional description
--kms-key-arn <kms-key-arn> KMS key to encrypt evaluation data at rest
-h, --help display help for command

Required-flag validation (captured via tui-harness, --name omitted):

$ agentcore eval batch-evaluation simulate --dataset /tmp/x.jsonl --runtime-id r --payload-template '{}' --evaluator E
Error: required option '--name <name>' not specified

[codecov-commenter]

Codecov Report

❌ Patch coverage is 74.42159% with 199 lines in your changes missing coverage. Please review.
✅ Project coverage is 96.12%. Comparing base (4d183dc) to head (80291f9).
⚠️ Report is 1 commits behind head on refactor.

Files with missing lines	Patch %	Lines
src/core/eval.tsx	65.23%	113 Missing ⚠️
src/core/eval/simulate.ts	4.70%	81 Missing ⚠️
src/handlers/eval/ondemand/evaluate/index.tsx	98.16%	2 Missing ⚠️
src/io/template.ts	91.30%	2 Missing ⚠️
src/core/invokeRuntime.ts	99.33%	1 Missing ⚠️

Additional details and impacted files

@@ Coverage Diff @@## refactor #2000 +/- ##
============================================
- Coverage 96.96% 96.12% -0.84% 
============================================
Files 364 370 +6 Lines 20758 21393 +635 ============================================
+ Hits 20127 20563 +436 - Misses 631 830 +199 

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[jariy17]

Superseded by #2032, which folds this replay path into the invokeDataset design (each example owns its invoker, passed into run()). Closing in favor of that.

[jariy17]

Summary

simulate replays a dataset through a runtime — one client-side invoke per scenario — then submits a StartBatchEvaluation over the sessions it created.
The service has no dataset data source, so the CLI creates the sessions and the service grades them.

CLI command structure

agentcore eval batch-evaluation
├── evaluate (existing)
├── simulate ← this PR
├── get
└── list

Flags (mirror runtime invoke):

Flag	Req	Notes
--runtime-id	✅	runtime to invoke per scenario
--qualifier		endpoint qualifier (default DEFAULT)
--payload-template '{"prompt":"{input}"}'	✅	{input} = scenario input
--header		ordered app header (repeatable)
--bearer-token		CUSTOM_JWT bearer token
--user-id		runtime user id
--dataset <path|id>	✅	local JSONL path or dataset id
--dataset-version		with a dataset id
--evaluator	✅	evaluator id(s)
--name	✅	unique in the account
--description		optional
--kms-key-arn		encrypt eval data at rest

content-type / accept are hardcoded application/json.

How it works

1. Load dataset — local JSONL path, or fetch by dataset id.
2. Invoke agent per scenario — concurrent, one fresh client-generated session id per scenario. Multi-turn = sequential invokes on one session.
3. Wait ~180s for span ingestion (spans must land before grading reads them).
4. StartBatchEvaluation — cloudWatchLogs scoped to those sessionIds + evaluationMetadata.sessionMetadata.
5. Poll with agentcore eval batch-evaluation get.

Design decisions

• Method on EvalClient — simulate is a method, no new class.
• Extracted invokeRuntime free fns to src/core/invokeRuntime.ts — no sibling core.runtime.* calls, star graph preserved; RuntimeClient.invokeRuntime becomes a 1-line delegate.
• Handler owns the public contract only — SimulateInput / SimulateResult are public; private helpers use local types (no handler-interface leaks inward).
• Generalized renderJsonTemplate in src/io — JSON-safe {key} substitution, not eval-specific.
• Session id client-generated — client-owned per AWS docs (min 33 chars; randomUUID() = 36), needed for microVM stickiness + multi-turn threading; matches the public bedrock_agentcore SDK rather than reading a response header.
• Ctrl-C cancellable — AbortSignal threaded through handler → core → invoke/sleep; aborts in-flight invokes, the dataset download, and the ingestion wait.

Testing

Live-verified in account 685197708687 / us-west-2, agent asdf_MyAgent-3s5axvBC6Q, evaluator Builtin.Helpfulness:

• Single-turn job sim_live_1f9e-cbcc074b38 → COMPLETED, 2/2 sessions graded.
• Multi-turn job sim_mt_2755-7517683065 (2-turn "remember 7 → recall it") → COMPLETED, 2/2 graded, 2 sessions scoped.

Automated:

• bun test src/ → 1383 pass; typecheck + oxlint clean.
• Unit tests: src/io/template.test.ts, src/handlers/eval/batch-evaluation/simulate/simulate.test.tsx.

--help (captured via tui-harness):

Usage: agentcore eval batch-evaluation simulate [options]
replay a dataset against a runtime, then batch-evaluate the resulting sessions
Options:
--runtime-id <runtime-id> runtime id to invoke per scenario
--qualifier <qualifier> runtime endpoint qualifier (default DEFAULT)
--payload-template <payload-template> JSON payload template; {input} is the scenario input, e.g.
{"prompt":"{input}"}
--header <header...> an ordered application header (repeatable)
--bearer-token <bearer-token> CUSTOM_JWT bearer token (for JWT-auth runtimes)
--user-id <user-id> runtime user id
--dataset <dataset> dataset source: local JSONL path or a dataset id
--dataset-version <dataset-version> dataset version (with a dataset id)
--evaluator <evaluator...> evaluator id(s) to apply
--name <name> batch evaluation name (unique in the account)
--description <description> optional description
--kms-key-arn <kms-key-arn> KMS key to encrypt evaluation data at rest
-h, --help display help for command

Required-flag validation (captured via tui-harness, --name omitted):

$ agentcore eval batch-evaluation simulate --dataset /tmp/x.jsonl --runtime-id r --payload-template '{}' --evaluator E
Error: required option '--name <name>' not specified

[codecov-commenter]

Codecov Report

❌ Patch coverage is 74.42159% with 199 lines in your changes missing coverage. Please review.
✅ Project coverage is 96.12%. Comparing base (4d183dc) to head (80291f9).
⚠️ Report is 1 commits behind head on refactor.

Files with missing lines	Patch %	Lines
src/core/eval.tsx	65.23%	113 Missing ⚠️
src/core/eval/simulate.ts	4.70%	81 Missing ⚠️
src/handlers/eval/ondemand/evaluate/index.tsx	98.16%	2 Missing ⚠️
src/io/template.ts	91.30%	2 Missing ⚠️
src/core/invokeRuntime.ts	99.33%	1 Missing ⚠️

Additional details and impacted files

@@ Coverage Diff @@## refactor #2000 +/- ##
============================================
- Coverage 96.96% 96.12% -0.84% 
============================================
Files 364 370 +6 Lines 20758 21393 +635 ============================================
+ Hits 20127 20563 +436 - Misses 631 830 +199 

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[jariy17]

Superseded by #2032, which folds this replay path into the invokeDataset design (each example owns its invoker, passed into run()). Closing in favor of that.