feat(project): add `project add evaluator code-based` by jariy17 · Pull Request #2144 · aws/agentcore-cli

jariy17 · 2026-08-28T22:14:37Z

What

Adds agentcore project add evaluator code-based — declarative code-based evaluators via projects — plus enables agentcore project remove evaluator. Follows the project add pattern (#2034 / #2004 / #2035 / #1998 / #2037) and, after review, mirrors the runtime layering (thin handler → templates/evaluator.ts owns the registry + buildManagedEvaluatorSpec, paralleling buildRuntimeSpec).

No mode/selector flag — the mode is inferred from what you pass (mirrors CodeBasedConfigSchema's managed XOR external):

You pass	Mode	Result
`--metric <library.Metric>`	managed · 3P	scaffolds a deepeval/autoevals Lambda from a template
(neither)	managed · empty	scaffolds an empty `@custom_code_based_evaluator()` stub you fill in
`--lambda-arn <arn>`	external	references an existing Lambda (no scaffold)

Command structure

agentcore project add evaluator add a custom evaluator to the current project
├── llm-as-a-judge existing — LLM prompted to score a session
└── code-based NEW — a Lambda that scores a session
agentcore project remove evaluator --name <name> NEW — enabled via the generic remove

agentcore project add evaluator code-based --help:

Usage: agentcore project add evaluator code-based [options]
add a code-based evaluator — a Lambda that scores a session. Pass a 3P metric,
an existing Lambda, or neither to scaffold an empty evaluator you fill in
Options:
--name <name> the name of the evaluator
--level <level> what to score: SESSION, TRACE, or TOOL_CALL
--metric <metric> 3P metric to scaffold as <library.Metric>,
e.g. deepeval.FaithfulnessMetric or autoevals.Factuality
--model <model> judge model for the 3P metric,
e.g. bedrock/anthropic.claude-3-5-sonnet-20240620-v1:0
--lambda-arn <lambda-arn> ARN of an existing Lambda that scores a session
--timeout-seconds <timeout-seconds> Lambda timeout in seconds (1-300)
--description <description> a description of what this evaluator measures
--kms-key-arn <kms-key-arn> customer-managed KMS key ARN to encrypt the evaluator
--tags <tags> tags to apply (JSON object of key/value strings)
-h, --help display help for command

Enforced in-handler (not shown by Commander): --name + --level required; exactly one of --metric/--lambda-arn; --metric must be <library>.<Class> where library ∈ {deepeval, autoevals} and Class is a single identifier; --model must be a Bedrock model id / inference-profile-or-foundation-model ARN (optionally bedrock/-prefixed) and requires --metric; --timeout-seconds/--model/--metric are managed-only. Managed auto-fills codeLocation=app/<name>, entrypoint=lambda_function.handler, per-library timeoutSeconds (deepeval 300, else 60), and additionalPolicies=["execution-role-policy.json"].

Commits

c4430c03 feat — the command + 3 scaffold templates + remove evaluator
baec1630 fix — guard app/<name> collisions (up-front, no partial writes)
e7bc3675 fix — validate --metric class + require a Bedrock --model
9bd79980 fix — echo the inferred mode + caveats at add time
32a10ef9 refactor — share toPythonPackageName via fsUtils; DEFAULT_TIMEOUT const
853dcf86 refactor — move template knowledge into templates/evaluator.ts (runtime layering)

Testing

bun run build OK · bun test src/handlers/project src/core/project → 597 pass / 0 fail.
Cloud bug bash (5 parallel agents; 2 deployed to a non-prod account, us-west-2, then tore down): all 5 flows + a 13-case error matrix pass at the CLI/scaffold/synth layer. The managed evaluator synthesizes correctly into AWS::BedrockAgentCore::Evaluator + Lambda + role + permissions, and generated Python ast.parses for deepeval + autoevals (bedrock + openai branches). Full report shared separately.

Known issues surfaced by the cloud deploy (both OUTSIDE this PR)

The evaluator authoring + synth work; end-to-end project deploy is currently blocked by two pre-existing bugs, neither in this feature's code:

Payments construct doesn't exist (feat(project): add payment resources #2120) — the vended src/assets/cdk/lib/cdk-stack.ts imports/instantiates AgentCorePayments, but no published @aws/agentcore-cdk version exports that name (alpha.49 and alpha.50 export AgentCorePaymentManager/Connector). So it's a template code bug, not a stale pin — a version bump can't fix it; the construct must be dropped or gated behind a spec payments-config check. tsc TS2305 → build/synth/deploy fail for every scaffolded project, evaluator or not. Owner: payments.
L3 evaluator "Access denied for Lambda" (reproducible) — with the payments line removed, synth emits the full evaluator resource set and deploy reaches AWS::BedrockAgentCore::Evaluator, which fails CREATE_FAILED: "Access denied for Lambda function …". Confirmed across two runs (not a race): the lambda:InvokeFunction/GetFunction grants to bedrock-agentcore.amazonaws.com reach CREATE_COMPLETE ~9s before the evaluator, yet the control-plane access check still denies — a deterministic authorization-shape mismatch (likely a required SourceAccount/SourceArn condition) in @aws/agentcore-cdk's AgentCoreEvaluator. Owner: agentcore-cdk L3.

🤖 Draft — CLI/authoring layer is complete and green; hold merge until the two deploy blockers (payments #2120, agentcore-cdk L3) land.

agentcore-devx-automation

AgentCore Harness Review

Verdict: Looks good

Small, focused fix that surfaces two real footguns of the code-based evaluator scaffold:

Empty stub silently returns Pass for every session (verified against src/assets/evaluators/python-lambda/lambda_function.py, which returns label="Pass").
Managed code-based evaluators aren't yet provisioned by project deploy.

Logic in index.ts (lines 158–166) matches the commit message: the "returns Pass" note is gated on !hasLambda && !hasMetric, and the "not yet provisioned" note is gated on !hasLambda, so --lambda-arn (external) correctly prints neither.

Tests in index.test.ts use real temp directories via mkdtemp and drive the handler through the router — no excessive mocking — and cover both the stub and external paths. Telemetry isn't warranted here since this only adds informational stderr output, not a new feature.

Nothing blocking.

codecov-commenter · 2026-08-28T22:18:03Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.31%. Comparing base (3d449c5) to head (9bd7998).

Additional details and impacted files

@@ Coverage Diff @@## refactor #2144 +/- ##
============================================
+ Coverage 97.29% 97.31% +0.01% 
============================================
Files 479 481 +2 Lines 29673 29865 +192 ============================================
+ Hits 28871 29063 +192 
Misses 802 802

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

agentcore-devx-automation · 2026-08-28T22:18:32Z

Claude Security Review: no high-confidence findings. (run)

agentcore-devx-automation · 2026-08-31T14:53:59Z

Claude Security Review: no high-confidence findings. (run)

jariy17 · 2026-08-31T17:24:00Z

+ const context: Record<string, unknown> = { Name: toPythonPackageName(flags["name"]) };
+
+ if (hasMetric) {
+ const raw = flags["metric"]!;


Can we use make this helper function and use zod here.

agentcore-devx-automation · 2026-08-31T17:28:09Z

Claude Security Review: no high-confidence findings. (run)

agentcore-devx-automation · 2026-08-31T17:31:40Z

Claude Security Review: no high-confidence findings. (run)

agentcore-devx-automation · 2026-08-31T17:35:47Z

Claude Security Review: no high-confidence findings. (run)

agentcore-devx-automation · 2026-08-31T17:53:03Z

Claude Security Review: no high-confidence findings. (run)

agentcore-devx-automation · 2026-08-31T18:20:07Z

Claude Security Review: no high-confidence findings. (run)

jariy17 · 2026-08-31T19:00:53Z

@@ -0,0 +1,15 @@
+{


I didn't want add an abstraction to generate common assets like this one due time constraints. We can look for this in the future.

agentcore-devx-automation · 2026-08-31T19:36:26Z

Claude Security Review: no high-confidence findings. (run)

Declarative code-based evaluators via projects. Mode is inferred from flags (mirrors CodeBasedConfigSchema managed XOR external): --lambda-arn -> external (BYO Lambda) --metric <library.Metric> -> managed 3P (deepeval/autoevals), scaffolded neither -> managed empty stub you fill in Scaffolds app/<name>/ from ported evaluator templates (python/deepeval/autoevals lambda), hardcodes codeLocation, and auto-wires additionalPolicies= [execution-role-policy.json]. Also enables `project remove evaluator`.

…aluators Runtimes, harnesses, and evaluators all scaffold into app/<name>, but the duplicate-name guard is per-resource-type and the tree write happens outside the rollback try/catch. An evaluator whose name matches an existing runtime/ harness dir (or a leftover from a removed evaluator) threw a raw 'File already exists' mid-write and orphaned partial files. Fail up front with a clear InputValidationError when app/<name> already exists.

…or code-based evaluators - Reject a namespaced/multi-dot metric class (e.g. deepeval.metrics.Faithfulness) that would render invalid Python; require a single class identifier. - --model is Bedrock-only: accept a bare model id / inference-profile-or- foundation-model ARN, optionally prefixed with bedrock/, validated via isValidBedrockModelId (same forms the llm-as-a-judge handler accepts). Non-Bedrock or slashless values now error instead of being silently dropped (deepeval) or passed to the wrong client (autoevals). - autoevals template prefixes bedrock/ for litellm routing now that Model is the bare id.

Print notes after add: the empty stub returns Pass for every session until implemented, and managed evaluators are scaffolded but not yet provisioned by 'project deploy' (no CDK/L3 support). External (--lambda-arn) prints neither.

…EOUT const; drop code comments

…templates layer Mirror the runtime layering: the handler now just parses/validates flags and passes a ManagedEvaluatorScaffoldInput; templates/evaluator.ts owns the library registry, per-library timeouts, render context, and buildManagedEvaluatorSpec (parallels buildRuntimeSpec). Also adds "evaluator" to RemoveResourceInput.

…aluators The L3 (@aws/agentcore-cdk) does provision spec.evaluators — synth emits AWS::BedrockAgentCore::Evaluator + Lambda — so the note was inaccurate. Keep the empty-stub 'returns Pass until implemented' note, which is still true.

agentcore-devx-automation · 2026-08-31T19:45:05Z

Claude Security Review: no high-confidence findings. (run)

jariy17 · 2026-08-28T22:14:37Z

What

Adds agentcore project add evaluator code-based — declarative code-based evaluators via projects — plus enables agentcore project remove evaluator. Follows the project add pattern (#2034 / #2004 / #2035 / #1998 / #2037) and, after review, mirrors the runtime layering (thin handler → templates/evaluator.ts owns the registry + buildManagedEvaluatorSpec, paralleling buildRuntimeSpec).

No mode/selector flag — the mode is inferred from what you pass (mirrors CodeBasedConfigSchema's managed XOR external):

You pass	Mode	Result
`--metric <library.Metric>`	managed · 3P	scaffolds a deepeval/autoevals Lambda from a template
(neither)	managed · empty	scaffolds an empty `@custom_code_based_evaluator()` stub you fill in
`--lambda-arn <arn>`	external	references an existing Lambda (no scaffold)

Command structure

agentcore project add evaluator add a custom evaluator to the current project
├── llm-as-a-judge existing — LLM prompted to score a session
└── code-based NEW — a Lambda that scores a session
agentcore project remove evaluator --name <name> NEW — enabled via the generic remove

agentcore project add evaluator code-based --help:

Usage: agentcore project add evaluator code-based [options]
add a code-based evaluator — a Lambda that scores a session. Pass a 3P metric,
an existing Lambda, or neither to scaffold an empty evaluator you fill in
Options:
--name <name> the name of the evaluator
--level <level> what to score: SESSION, TRACE, or TOOL_CALL
--metric <metric> 3P metric to scaffold as <library.Metric>,
e.g. deepeval.FaithfulnessMetric or autoevals.Factuality
--model <model> judge model for the 3P metric,
e.g. bedrock/anthropic.claude-3-5-sonnet-20240620-v1:0
--lambda-arn <lambda-arn> ARN of an existing Lambda that scores a session
--timeout-seconds <timeout-seconds> Lambda timeout in seconds (1-300)
--description <description> a description of what this evaluator measures
--kms-key-arn <kms-key-arn> customer-managed KMS key ARN to encrypt the evaluator
--tags <tags> tags to apply (JSON object of key/value strings)
-h, --help display help for command

Enforced in-handler (not shown by Commander): --name + --level required; exactly one of --metric/--lambda-arn; --metric must be <library>.<Class> where library ∈ {deepeval, autoevals} and Class is a single identifier; --model must be a Bedrock model id / inference-profile-or-foundation-model ARN (optionally bedrock/-prefixed) and requires --metric; --timeout-seconds/--model/--metric are managed-only. Managed auto-fills codeLocation=app/<name>, entrypoint=lambda_function.handler, per-library timeoutSeconds (deepeval 300, else 60), and additionalPolicies=["execution-role-policy.json"].

Commits

c4430c03 feat — the command + 3 scaffold templates + remove evaluator
baec1630 fix — guard app/<name> collisions (up-front, no partial writes)
e7bc3675 fix — validate --metric class + require a Bedrock --model
9bd79980 fix — echo the inferred mode + caveats at add time
32a10ef9 refactor — share toPythonPackageName via fsUtils; DEFAULT_TIMEOUT const
853dcf86 refactor — move template knowledge into templates/evaluator.ts (runtime layering)

Testing

bun run build OK · bun test src/handlers/project src/core/project → 597 pass / 0 fail.
Cloud bug bash (5 parallel agents; 2 deployed to a non-prod account, us-west-2, then tore down): all 5 flows + a 13-case error matrix pass at the CLI/scaffold/synth layer. The managed evaluator synthesizes correctly into AWS::BedrockAgentCore::Evaluator + Lambda + role + permissions, and generated Python ast.parses for deepeval + autoevals (bedrock + openai branches). Full report shared separately.

Known issues surfaced by the cloud deploy (both OUTSIDE this PR)

The evaluator authoring + synth work; end-to-end project deploy is currently blocked by two pre-existing bugs, neither in this feature's code:

Payments construct doesn't exist (feat(project): add payment resources #2120) — the vended src/assets/cdk/lib/cdk-stack.ts imports/instantiates AgentCorePayments, but no published @aws/agentcore-cdk version exports that name (alpha.49 and alpha.50 export AgentCorePaymentManager/Connector). So it's a template code bug, not a stale pin — a version bump can't fix it; the construct must be dropped or gated behind a spec payments-config check. tsc TS2305 → build/synth/deploy fail for every scaffolded project, evaluator or not. Owner: payments.
L3 evaluator "Access denied for Lambda" (reproducible) — with the payments line removed, synth emits the full evaluator resource set and deploy reaches AWS::BedrockAgentCore::Evaluator, which fails CREATE_FAILED: "Access denied for Lambda function …". Confirmed across two runs (not a race): the lambda:InvokeFunction/GetFunction grants to bedrock-agentcore.amazonaws.com reach CREATE_COMPLETE ~9s before the evaluator, yet the control-plane access check still denies — a deterministic authorization-shape mismatch (likely a required SourceAccount/SourceArn condition) in @aws/agentcore-cdk's AgentCoreEvaluator. Owner: agentcore-cdk L3.

🤖 Draft — CLI/authoring layer is complete and green; hold merge until the two deploy blockers (payments #2120, agentcore-cdk L3) land.

agentcore-devx-automation

AgentCore Harness Review

Verdict: Looks good

Small, focused fix that surfaces two real footguns of the code-based evaluator scaffold:

Empty stub silently returns Pass for every session (verified against src/assets/evaluators/python-lambda/lambda_function.py, which returns label="Pass").
Managed code-based evaluators aren't yet provisioned by project deploy.

Logic in index.ts (lines 158–166) matches the commit message: the "returns Pass" note is gated on !hasLambda && !hasMetric, and the "not yet provisioned" note is gated on !hasLambda, so --lambda-arn (external) correctly prints neither.

Tests in index.test.ts use real temp directories via mkdtemp and drive the handler through the router — no excessive mocking — and cover both the stub and external paths. Telemetry isn't warranted here since this only adds informational stderr output, not a new feature.

Nothing blocking.

codecov-commenter · 2026-08-28T22:18:03Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.31%. Comparing base (3d449c5) to head (9bd7998).

Additional details and impacted files

@@ Coverage Diff @@## refactor #2144 +/- ##
============================================
+ Coverage 97.29% 97.31% +0.01% 
============================================
Files 479 481 +2 Lines 29673 29865 +192 ============================================
+ Hits 28871 29063 +192 
Misses 802 802

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

agentcore-devx-automation · 2026-08-28T22:18:32Z

Claude Security Review: no high-confidence findings. (run)

agentcore-devx-automation · 2026-08-31T14:53:59Z

Claude Security Review: no high-confidence findings. (run)

jariy17 · 2026-08-31T17:24:00Z

+ const context: Record<string, unknown> = { Name: toPythonPackageName(flags["name"]) };
+
+ if (hasMetric) {
+ const raw = flags["metric"]!;


Can we use make this helper function and use zod here.

agentcore-devx-automation · 2026-08-31T17:28:09Z

Claude Security Review: no high-confidence findings. (run)

agentcore-devx-automation · 2026-08-31T17:31:40Z

Claude Security Review: no high-confidence findings. (run)

agentcore-devx-automation · 2026-08-31T17:35:47Z

Claude Security Review: no high-confidence findings. (run)

agentcore-devx-automation · 2026-08-31T17:53:03Z

Claude Security Review: no high-confidence findings. (run)

agentcore-devx-automation · 2026-08-31T18:20:07Z

Claude Security Review: no high-confidence findings. (run)

jariy17 · 2026-08-31T19:00:53Z

@@ -0,0 +1,15 @@
+{


I didn't want add an abstraction to generate common assets like this one due time constraints. We can look for this in the future.

agentcore-devx-automation · 2026-08-31T19:36:26Z

Claude Security Review: no high-confidence findings. (run)

Declarative code-based evaluators via projects. Mode is inferred from flags (mirrors CodeBasedConfigSchema managed XOR external): --lambda-arn -> external (BYO Lambda) --metric <library.Metric> -> managed 3P (deepeval/autoevals), scaffolded neither -> managed empty stub you fill in Scaffolds app/<name>/ from ported evaluator templates (python/deepeval/autoevals lambda), hardcodes codeLocation, and auto-wires additionalPolicies= [execution-role-policy.json]. Also enables `project remove evaluator`.

…aluators Runtimes, harnesses, and evaluators all scaffold into app/<name>, but the duplicate-name guard is per-resource-type and the tree write happens outside the rollback try/catch. An evaluator whose name matches an existing runtime/ harness dir (or a leftover from a removed evaluator) threw a raw 'File already exists' mid-write and orphaned partial files. Fail up front with a clear InputValidationError when app/<name> already exists.

…or code-based evaluators - Reject a namespaced/multi-dot metric class (e.g. deepeval.metrics.Faithfulness) that would render invalid Python; require a single class identifier. - --model is Bedrock-only: accept a bare model id / inference-profile-or- foundation-model ARN, optionally prefixed with bedrock/, validated via isValidBedrockModelId (same forms the llm-as-a-judge handler accepts). Non-Bedrock or slashless values now error instead of being silently dropped (deepeval) or passed to the wrong client (autoevals). - autoevals template prefixes bedrock/ for litellm routing now that Model is the bare id.

Print notes after add: the empty stub returns Pass for every session until implemented, and managed evaluators are scaffolded but not yet provisioned by 'project deploy' (no CDK/L3 support). External (--lambda-arn) prints neither.

…EOUT const; drop code comments

…templates layer Mirror the runtime layering: the handler now just parses/validates flags and passes a ManagedEvaluatorScaffoldInput; templates/evaluator.ts owns the library registry, per-library timeouts, render context, and buildManagedEvaluatorSpec (parallels buildRuntimeSpec). Also adds "evaluator" to RemoveResourceInput.

…aluators The L3 (@aws/agentcore-cdk) does provision spec.evaluators — synth emits AWS::BedrockAgentCore::Evaluator + Lambda — so the note was inaccurate. Keep the empty-stub 'returns Pass until implemented' note, which is still true.

agentcore-devx-automation · 2026-08-31T19:45:05Z

Claude Security Review: no high-confidence findings. (run)

jariy17 · 2026-08-28T22:14:37Z

What

Adds agentcore project add evaluator code-based — declarative code-based evaluators via projects — plus enables agentcore project remove evaluator. Follows the project add pattern (#2034 / #2004 / #2035 / #1998 / #2037) and, after review, mirrors the runtime layering (thin handler → templates/evaluator.ts owns the registry + buildManagedEvaluatorSpec, paralleling buildRuntimeSpec).

No mode/selector flag — the mode is inferred from what you pass (mirrors CodeBasedConfigSchema's managed XOR external):

You pass	Mode	Result
`--metric <library.Metric>`	managed · 3P	scaffolds a deepeval/autoevals Lambda from a template
(neither)	managed · empty	scaffolds an empty `@custom_code_based_evaluator()` stub you fill in
`--lambda-arn <arn>`	external	references an existing Lambda (no scaffold)

Command structure

agentcore project add evaluator add a custom evaluator to the current project
├── llm-as-a-judge existing — LLM prompted to score a session
└── code-based NEW — a Lambda that scores a session
agentcore project remove evaluator --name <name> NEW — enabled via the generic remove

agentcore project add evaluator code-based --help:

Usage: agentcore project add evaluator code-based [options]
add a code-based evaluator — a Lambda that scores a session. Pass a 3P metric,
an existing Lambda, or neither to scaffold an empty evaluator you fill in
Options:
--name <name> the name of the evaluator
--level <level> what to score: SESSION, TRACE, or TOOL_CALL
--metric <metric> 3P metric to scaffold as <library.Metric>,
e.g. deepeval.FaithfulnessMetric or autoevals.Factuality
--model <model> judge model for the 3P metric,
e.g. bedrock/anthropic.claude-3-5-sonnet-20240620-v1:0
--lambda-arn <lambda-arn> ARN of an existing Lambda that scores a session
--timeout-seconds <timeout-seconds> Lambda timeout in seconds (1-300)
--description <description> a description of what this evaluator measures
--kms-key-arn <kms-key-arn> customer-managed KMS key ARN to encrypt the evaluator
--tags <tags> tags to apply (JSON object of key/value strings)
-h, --help display help for command

Enforced in-handler (not shown by Commander): --name + --level required; exactly one of --metric/--lambda-arn; --metric must be <library>.<Class> where library ∈ {deepeval, autoevals} and Class is a single identifier; --model must be a Bedrock model id / inference-profile-or-foundation-model ARN (optionally bedrock/-prefixed) and requires --metric; --timeout-seconds/--model/--metric are managed-only. Managed auto-fills codeLocation=app/<name>, entrypoint=lambda_function.handler, per-library timeoutSeconds (deepeval 300, else 60), and additionalPolicies=["execution-role-policy.json"].

Commits

c4430c03 feat — the command + 3 scaffold templates + remove evaluator
baec1630 fix — guard app/<name> collisions (up-front, no partial writes)
e7bc3675 fix — validate --metric class + require a Bedrock --model
9bd79980 fix — echo the inferred mode + caveats at add time
32a10ef9 refactor — share toPythonPackageName via fsUtils; DEFAULT_TIMEOUT const
853dcf86 refactor — move template knowledge into templates/evaluator.ts (runtime layering)

Testing

bun run build OK · bun test src/handlers/project src/core/project → 597 pass / 0 fail.
Cloud bug bash (5 parallel agents; 2 deployed to a non-prod account, us-west-2, then tore down): all 5 flows + a 13-case error matrix pass at the CLI/scaffold/synth layer. The managed evaluator synthesizes correctly into AWS::BedrockAgentCore::Evaluator + Lambda + role + permissions, and generated Python ast.parses for deepeval + autoevals (bedrock + openai branches). Full report shared separately.

Known issues surfaced by the cloud deploy (both OUTSIDE this PR)

The evaluator authoring + synth work; end-to-end project deploy is currently blocked by two pre-existing bugs, neither in this feature's code:

Payments construct doesn't exist (feat(project): add payment resources #2120) — the vended src/assets/cdk/lib/cdk-stack.ts imports/instantiates AgentCorePayments, but no published @aws/agentcore-cdk version exports that name (alpha.49 and alpha.50 export AgentCorePaymentManager/Connector). So it's a template code bug, not a stale pin — a version bump can't fix it; the construct must be dropped or gated behind a spec payments-config check. tsc TS2305 → build/synth/deploy fail for every scaffolded project, evaluator or not. Owner: payments.
L3 evaluator "Access denied for Lambda" (reproducible) — with the payments line removed, synth emits the full evaluator resource set and deploy reaches AWS::BedrockAgentCore::Evaluator, which fails CREATE_FAILED: "Access denied for Lambda function …". Confirmed across two runs (not a race): the lambda:InvokeFunction/GetFunction grants to bedrock-agentcore.amazonaws.com reach CREATE_COMPLETE ~9s before the evaluator, yet the control-plane access check still denies — a deterministic authorization-shape mismatch (likely a required SourceAccount/SourceArn condition) in @aws/agentcore-cdk's AgentCoreEvaluator. Owner: agentcore-cdk L3.

🤖 Draft — CLI/authoring layer is complete and green; hold merge until the two deploy blockers (payments #2120, agentcore-cdk L3) land.

agentcore-devx-automation

AgentCore Harness Review

Verdict: Looks good

Small, focused fix that surfaces two real footguns of the code-based evaluator scaffold:

Empty stub silently returns Pass for every session (verified against src/assets/evaluators/python-lambda/lambda_function.py, which returns label="Pass").
Managed code-based evaluators aren't yet provisioned by project deploy.

Logic in index.ts (lines 158–166) matches the commit message: the "returns Pass" note is gated on !hasLambda && !hasMetric, and the "not yet provisioned" note is gated on !hasLambda, so --lambda-arn (external) correctly prints neither.

Tests in index.test.ts use real temp directories via mkdtemp and drive the handler through the router — no excessive mocking — and cover both the stub and external paths. Telemetry isn't warranted here since this only adds informational stderr output, not a new feature.

Nothing blocking.

codecov-commenter · 2026-08-28T22:18:03Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.31%. Comparing base (3d449c5) to head (9bd7998).

Additional details and impacted files

@@ Coverage Diff @@## refactor #2144 +/- ##
============================================
+ Coverage 97.29% 97.31% +0.01% 
============================================
Files 479 481 +2 Lines 29673 29865 +192 ============================================
+ Hits 28871 29063 +192 
Misses 802 802

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

agentcore-devx-automation · 2026-08-28T22:18:32Z

Claude Security Review: no high-confidence findings. (run)

agentcore-devx-automation · 2026-08-31T14:53:59Z

Claude Security Review: no high-confidence findings. (run)

jariy17 · 2026-08-31T17:24:00Z

+ const context: Record<string, unknown> = { Name: toPythonPackageName(flags["name"]) };
+
+ if (hasMetric) {
+ const raw = flags["metric"]!;


Can we use make this helper function and use zod here.

agentcore-devx-automation · 2026-08-31T17:28:09Z

Claude Security Review: no high-confidence findings. (run)

agentcore-devx-automation · 2026-08-31T17:31:40Z

Claude Security Review: no high-confidence findings. (run)

agentcore-devx-automation · 2026-08-31T17:35:47Z

Claude Security Review: no high-confidence findings. (run)

agentcore-devx-automation · 2026-08-31T17:53:03Z

Claude Security Review: no high-confidence findings. (run)

agentcore-devx-automation · 2026-08-31T18:20:07Z

Claude Security Review: no high-confidence findings. (run)

jariy17 · 2026-08-31T19:00:53Z

@@ -0,0 +1,15 @@
+{


I didn't want add an abstraction to generate common assets like this one due time constraints. We can look for this in the future.

agentcore-devx-automation · 2026-08-31T19:36:26Z

Claude Security Review: no high-confidence findings. (run)

Declarative code-based evaluators via projects. Mode is inferred from flags (mirrors CodeBasedConfigSchema managed XOR external): --lambda-arn -> external (BYO Lambda) --metric <library.Metric> -> managed 3P (deepeval/autoevals), scaffolded neither -> managed empty stub you fill in Scaffolds app/<name>/ from ported evaluator templates (python/deepeval/autoevals lambda), hardcodes codeLocation, and auto-wires additionalPolicies= [execution-role-policy.json]. Also enables `project remove evaluator`.

…aluators Runtimes, harnesses, and evaluators all scaffold into app/<name>, but the duplicate-name guard is per-resource-type and the tree write happens outside the rollback try/catch. An evaluator whose name matches an existing runtime/ harness dir (or a leftover from a removed evaluator) threw a raw 'File already exists' mid-write and orphaned partial files. Fail up front with a clear InputValidationError when app/<name> already exists.

…or code-based evaluators - Reject a namespaced/multi-dot metric class (e.g. deepeval.metrics.Faithfulness) that would render invalid Python; require a single class identifier. - --model is Bedrock-only: accept a bare model id / inference-profile-or- foundation-model ARN, optionally prefixed with bedrock/, validated via isValidBedrockModelId (same forms the llm-as-a-judge handler accepts). Non-Bedrock or slashless values now error instead of being silently dropped (deepeval) or passed to the wrong client (autoevals). - autoevals template prefixes bedrock/ for litellm routing now that Model is the bare id.

Print notes after add: the empty stub returns Pass for every session until implemented, and managed evaluators are scaffolded but not yet provisioned by 'project deploy' (no CDK/L3 support). External (--lambda-arn) prints neither.

…EOUT const; drop code comments

…templates layer Mirror the runtime layering: the handler now just parses/validates flags and passes a ManagedEvaluatorScaffoldInput; templates/evaluator.ts owns the library registry, per-library timeouts, render context, and buildManagedEvaluatorSpec (parallels buildRuntimeSpec). Also adds "evaluator" to RemoveResourceInput.

…aluators The L3 (@aws/agentcore-cdk) does provision spec.evaluators — synth emits AWS::BedrockAgentCore::Evaluator + Lambda — so the note was inaccurate. Keep the empty-stub 'returns Pass until implemented' note, which is still true.

agentcore-devx-automation · 2026-08-31T19:45:05Z

Claude Security Review: no high-confidence findings. (run)

jariy17 · 2026-08-28T22:14:37Z

What

Adds agentcore project add evaluator code-based — declarative code-based evaluators via projects — plus enables agentcore project remove evaluator. Follows the project add pattern (#2034 / #2004 / #2035 / #1998 / #2037) and, after review, mirrors the runtime layering (thin handler → templates/evaluator.ts owns the registry + buildManagedEvaluatorSpec, paralleling buildRuntimeSpec).

No mode/selector flag — the mode is inferred from what you pass (mirrors CodeBasedConfigSchema's managed XOR external):

You pass	Mode	Result
`--metric <library.Metric>`	managed · 3P	scaffolds a deepeval/autoevals Lambda from a template
(neither)	managed · empty	scaffolds an empty `@custom_code_based_evaluator()` stub you fill in
`--lambda-arn <arn>`	external	references an existing Lambda (no scaffold)

Command structure

agentcore project add evaluator add a custom evaluator to the current project
├── llm-as-a-judge existing — LLM prompted to score a session
└── code-based NEW — a Lambda that scores a session
agentcore project remove evaluator --name <name> NEW — enabled via the generic remove

agentcore project add evaluator code-based --help:

Usage: agentcore project add evaluator code-based [options]
add a code-based evaluator — a Lambda that scores a session. Pass a 3P metric,
an existing Lambda, or neither to scaffold an empty evaluator you fill in
Options:
--name <name> the name of the evaluator
--level <level> what to score: SESSION, TRACE, or TOOL_CALL
--metric <metric> 3P metric to scaffold as <library.Metric>,
e.g. deepeval.FaithfulnessMetric or autoevals.Factuality
--model <model> judge model for the 3P metric,
e.g. bedrock/anthropic.claude-3-5-sonnet-20240620-v1:0
--lambda-arn <lambda-arn> ARN of an existing Lambda that scores a session
--timeout-seconds <timeout-seconds> Lambda timeout in seconds (1-300)
--description <description> a description of what this evaluator measures
--kms-key-arn <kms-key-arn> customer-managed KMS key ARN to encrypt the evaluator
--tags <tags> tags to apply (JSON object of key/value strings)
-h, --help display help for command

Enforced in-handler (not shown by Commander): --name + --level required; exactly one of --metric/--lambda-arn; --metric must be <library>.<Class> where library ∈ {deepeval, autoevals} and Class is a single identifier; --model must be a Bedrock model id / inference-profile-or-foundation-model ARN (optionally bedrock/-prefixed) and requires --metric; --timeout-seconds/--model/--metric are managed-only. Managed auto-fills codeLocation=app/<name>, entrypoint=lambda_function.handler, per-library timeoutSeconds (deepeval 300, else 60), and additionalPolicies=["execution-role-policy.json"].

Commits

c4430c03 feat — the command + 3 scaffold templates + remove evaluator
baec1630 fix — guard app/<name> collisions (up-front, no partial writes)
e7bc3675 fix — validate --metric class + require a Bedrock --model
9bd79980 fix — echo the inferred mode + caveats at add time
32a10ef9 refactor — share toPythonPackageName via fsUtils; DEFAULT_TIMEOUT const
853dcf86 refactor — move template knowledge into templates/evaluator.ts (runtime layering)

Testing

bun run build OK · bun test src/handlers/project src/core/project → 597 pass / 0 fail.
Cloud bug bash (5 parallel agents; 2 deployed to a non-prod account, us-west-2, then tore down): all 5 flows + a 13-case error matrix pass at the CLI/scaffold/synth layer. The managed evaluator synthesizes correctly into AWS::BedrockAgentCore::Evaluator + Lambda + role + permissions, and generated Python ast.parses for deepeval + autoevals (bedrock + openai branches). Full report shared separately.

Known issues surfaced by the cloud deploy (both OUTSIDE this PR)

The evaluator authoring + synth work; end-to-end project deploy is currently blocked by two pre-existing bugs, neither in this feature's code:

Payments construct doesn't exist (feat(project): add payment resources #2120) — the vended src/assets/cdk/lib/cdk-stack.ts imports/instantiates AgentCorePayments, but no published @aws/agentcore-cdk version exports that name (alpha.49 and alpha.50 export AgentCorePaymentManager/Connector). So it's a template code bug, not a stale pin — a version bump can't fix it; the construct must be dropped or gated behind a spec payments-config check. tsc TS2305 → build/synth/deploy fail for every scaffolded project, evaluator or not. Owner: payments.
L3 evaluator "Access denied for Lambda" (reproducible) — with the payments line removed, synth emits the full evaluator resource set and deploy reaches AWS::BedrockAgentCore::Evaluator, which fails CREATE_FAILED: "Access denied for Lambda function …". Confirmed across two runs (not a race): the lambda:InvokeFunction/GetFunction grants to bedrock-agentcore.amazonaws.com reach CREATE_COMPLETE ~9s before the evaluator, yet the control-plane access check still denies — a deterministic authorization-shape mismatch (likely a required SourceAccount/SourceArn condition) in @aws/agentcore-cdk's AgentCoreEvaluator. Owner: agentcore-cdk L3.

🤖 Draft — CLI/authoring layer is complete and green; hold merge until the two deploy blockers (payments #2120, agentcore-cdk L3) land.

agentcore-devx-automation

AgentCore Harness Review

Verdict: Looks good

Small, focused fix that surfaces two real footguns of the code-based evaluator scaffold:

Empty stub silently returns Pass for every session (verified against src/assets/evaluators/python-lambda/lambda_function.py, which returns label="Pass").
Managed code-based evaluators aren't yet provisioned by project deploy.

Logic in index.ts (lines 158–166) matches the commit message: the "returns Pass" note is gated on !hasLambda && !hasMetric, and the "not yet provisioned" note is gated on !hasLambda, so --lambda-arn (external) correctly prints neither.

Tests in index.test.ts use real temp directories via mkdtemp and drive the handler through the router — no excessive mocking — and cover both the stub and external paths. Telemetry isn't warranted here since this only adds informational stderr output, not a new feature.

Nothing blocking.

codecov-commenter · 2026-08-28T22:18:03Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.31%. Comparing base (3d449c5) to head (9bd7998).

Additional details and impacted files

@@ Coverage Diff @@## refactor #2144 +/- ##
============================================
+ Coverage 97.29% 97.31% +0.01% 
============================================
Files 479 481 +2 Lines 29673 29865 +192 ============================================
+ Hits 28871 29063 +192 
Misses 802 802

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

agentcore-devx-automation · 2026-08-28T22:18:32Z

Claude Security Review: no high-confidence findings. (run)

agentcore-devx-automation · 2026-08-31T14:53:59Z

Claude Security Review: no high-confidence findings. (run)

jariy17 · 2026-08-31T17:24:00Z

+ const context: Record<string, unknown> = { Name: toPythonPackageName(flags["name"]) };
+
+ if (hasMetric) {
+ const raw = flags["metric"]!;


Can we use make this helper function and use zod here.

agentcore-devx-automation · 2026-08-31T17:28:09Z

Claude Security Review: no high-confidence findings. (run)

agentcore-devx-automation · 2026-08-31T17:31:40Z

Claude Security Review: no high-confidence findings. (run)

agentcore-devx-automation · 2026-08-31T17:35:47Z

Claude Security Review: no high-confidence findings. (run)

agentcore-devx-automation · 2026-08-31T17:53:03Z

Claude Security Review: no high-confidence findings. (run)

agentcore-devx-automation · 2026-08-31T18:20:07Z

Claude Security Review: no high-confidence findings. (run)

jariy17 · 2026-08-31T19:00:53Z

@@ -0,0 +1,15 @@
+{


I didn't want add an abstraction to generate common assets like this one due time constraints. We can look for this in the future.

agentcore-devx-automation · 2026-08-31T19:36:26Z

Claude Security Review: no high-confidence findings. (run)

Declarative code-based evaluators via projects. Mode is inferred from flags (mirrors CodeBasedConfigSchema managed XOR external): --lambda-arn -> external (BYO Lambda) --metric <library.Metric> -> managed 3P (deepeval/autoevals), scaffolded neither -> managed empty stub you fill in Scaffolds app/<name>/ from ported evaluator templates (python/deepeval/autoevals lambda), hardcodes codeLocation, and auto-wires additionalPolicies= [execution-role-policy.json]. Also enables `project remove evaluator`.

…aluators Runtimes, harnesses, and evaluators all scaffold into app/<name>, but the duplicate-name guard is per-resource-type and the tree write happens outside the rollback try/catch. An evaluator whose name matches an existing runtime/ harness dir (or a leftover from a removed evaluator) threw a raw 'File already exists' mid-write and orphaned partial files. Fail up front with a clear InputValidationError when app/<name> already exists.

…or code-based evaluators - Reject a namespaced/multi-dot metric class (e.g. deepeval.metrics.Faithfulness) that would render invalid Python; require a single class identifier. - --model is Bedrock-only: accept a bare model id / inference-profile-or- foundation-model ARN, optionally prefixed with bedrock/, validated via isValidBedrockModelId (same forms the llm-as-a-judge handler accepts). Non-Bedrock or slashless values now error instead of being silently dropped (deepeval) or passed to the wrong client (autoevals). - autoevals template prefixes bedrock/ for litellm routing now that Model is the bare id.

Print notes after add: the empty stub returns Pass for every session until implemented, and managed evaluators are scaffolded but not yet provisioned by 'project deploy' (no CDK/L3 support). External (--lambda-arn) prints neither.

…EOUT const; drop code comments

…templates layer Mirror the runtime layering: the handler now just parses/validates flags and passes a ManagedEvaluatorScaffoldInput; templates/evaluator.ts owns the library registry, per-library timeouts, render context, and buildManagedEvaluatorSpec (parallels buildRuntimeSpec). Also adds "evaluator" to RemoveResourceInput.

…aluators The L3 (@aws/agentcore-cdk) does provision spec.evaluators — synth emits AWS::BedrockAgentCore::Evaluator + Lambda — so the note was inaccurate. Keep the empty-stub 'returns Pass until implemented' note, which is still true.

agentcore-devx-automation · 2026-08-31T19:45:05Z

Claude Security Review: no high-confidence findings. (run)

jariy17 · 2026-08-28T22:14:37Z

What

Adds agentcore project add evaluator code-based — declarative code-based evaluators via projects — plus enables agentcore project remove evaluator. Follows the project add pattern (#2034 / #2004 / #2035 / #1998 / #2037) and, after review, mirrors the runtime layering (thin handler → templates/evaluator.ts owns the registry + buildManagedEvaluatorSpec, paralleling buildRuntimeSpec).

No mode/selector flag — the mode is inferred from what you pass (mirrors CodeBasedConfigSchema's managed XOR external):

You pass	Mode	Result
`--metric <library.Metric>`	managed · 3P	scaffolds a deepeval/autoevals Lambda from a template
(neither)	managed · empty	scaffolds an empty `@custom_code_based_evaluator()` stub you fill in
`--lambda-arn <arn>`	external	references an existing Lambda (no scaffold)

Command structure

agentcore project add evaluator add a custom evaluator to the current project
├── llm-as-a-judge existing — LLM prompted to score a session
└── code-based NEW — a Lambda that scores a session
agentcore project remove evaluator --name <name> NEW — enabled via the generic remove

agentcore project add evaluator code-based --help:

Usage: agentcore project add evaluator code-based [options]
add a code-based evaluator — a Lambda that scores a session. Pass a 3P metric,
an existing Lambda, or neither to scaffold an empty evaluator you fill in
Options:
--name <name> the name of the evaluator
--level <level> what to score: SESSION, TRACE, or TOOL_CALL
--metric <metric> 3P metric to scaffold as <library.Metric>,
e.g. deepeval.FaithfulnessMetric or autoevals.Factuality
--model <model> judge model for the 3P metric,
e.g. bedrock/anthropic.claude-3-5-sonnet-20240620-v1:0
--lambda-arn <lambda-arn> ARN of an existing Lambda that scores a session
--timeout-seconds <timeout-seconds> Lambda timeout in seconds (1-300)
--description <description> a description of what this evaluator measures
--kms-key-arn <kms-key-arn> customer-managed KMS key ARN to encrypt the evaluator
--tags <tags> tags to apply (JSON object of key/value strings)
-h, --help display help for command

Enforced in-handler (not shown by Commander): --name + --level required; exactly one of --metric/--lambda-arn; --metric must be <library>.<Class> where library ∈ {deepeval, autoevals} and Class is a single identifier; --model must be a Bedrock model id / inference-profile-or-foundation-model ARN (optionally bedrock/-prefixed) and requires --metric; --timeout-seconds/--model/--metric are managed-only. Managed auto-fills codeLocation=app/<name>, entrypoint=lambda_function.handler, per-library timeoutSeconds (deepeval 300, else 60), and additionalPolicies=["execution-role-policy.json"].

Commits

c4430c03 feat — the command + 3 scaffold templates + remove evaluator
baec1630 fix — guard app/<name> collisions (up-front, no partial writes)
e7bc3675 fix — validate --metric class + require a Bedrock --model
9bd79980 fix — echo the inferred mode + caveats at add time
32a10ef9 refactor — share toPythonPackageName via fsUtils; DEFAULT_TIMEOUT const
853dcf86 refactor — move template knowledge into templates/evaluator.ts (runtime layering)

Testing

bun run build OK · bun test src/handlers/project src/core/project → 597 pass / 0 fail.
Cloud bug bash (5 parallel agents; 2 deployed to a non-prod account, us-west-2, then tore down): all 5 flows + a 13-case error matrix pass at the CLI/scaffold/synth layer. The managed evaluator synthesizes correctly into AWS::BedrockAgentCore::Evaluator + Lambda + role + permissions, and generated Python ast.parses for deepeval + autoevals (bedrock + openai branches). Full report shared separately.

Known issues surfaced by the cloud deploy (both OUTSIDE this PR)

The evaluator authoring + synth work; end-to-end project deploy is currently blocked by two pre-existing bugs, neither in this feature's code:

Payments construct doesn't exist (feat(project): add payment resources #2120) — the vended src/assets/cdk/lib/cdk-stack.ts imports/instantiates AgentCorePayments, but no published @aws/agentcore-cdk version exports that name (alpha.49 and alpha.50 export AgentCorePaymentManager/Connector). So it's a template code bug, not a stale pin — a version bump can't fix it; the construct must be dropped or gated behind a spec payments-config check. tsc TS2305 → build/synth/deploy fail for every scaffolded project, evaluator or not. Owner: payments.
L3 evaluator "Access denied for Lambda" (reproducible) — with the payments line removed, synth emits the full evaluator resource set and deploy reaches AWS::BedrockAgentCore::Evaluator, which fails CREATE_FAILED: "Access denied for Lambda function …". Confirmed across two runs (not a race): the lambda:InvokeFunction/GetFunction grants to bedrock-agentcore.amazonaws.com reach CREATE_COMPLETE ~9s before the evaluator, yet the control-plane access check still denies — a deterministic authorization-shape mismatch (likely a required SourceAccount/SourceArn condition) in @aws/agentcore-cdk's AgentCoreEvaluator. Owner: agentcore-cdk L3.

🤖 Draft — CLI/authoring layer is complete and green; hold merge until the two deploy blockers (payments #2120, agentcore-cdk L3) land.

agentcore-devx-automation

AgentCore Harness Review

Verdict: Looks good

Small, focused fix that surfaces two real footguns of the code-based evaluator scaffold:

Empty stub silently returns Pass for every session (verified against src/assets/evaluators/python-lambda/lambda_function.py, which returns label="Pass").
Managed code-based evaluators aren't yet provisioned by project deploy.

Logic in index.ts (lines 158–166) matches the commit message: the "returns Pass" note is gated on !hasLambda && !hasMetric, and the "not yet provisioned" note is gated on !hasLambda, so --lambda-arn (external) correctly prints neither.

Tests in index.test.ts use real temp directories via mkdtemp and drive the handler through the router — no excessive mocking — and cover both the stub and external paths. Telemetry isn't warranted here since this only adds informational stderr output, not a new feature.

Nothing blocking.

codecov-commenter · 2026-08-28T22:18:03Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.31%. Comparing base (3d449c5) to head (9bd7998).

Additional details and impacted files

@@ Coverage Diff @@## refactor #2144 +/- ##
============================================
+ Coverage 97.29% 97.31% +0.01% 
============================================
Files 479 481 +2 Lines 29673 29865 +192 ============================================
+ Hits 28871 29063 +192 
Misses 802 802

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

agentcore-devx-automation · 2026-08-28T22:18:32Z

Claude Security Review: no high-confidence findings. (run)

agentcore-devx-automation · 2026-08-31T14:53:59Z

Claude Security Review: no high-confidence findings. (run)

jariy17 · 2026-08-31T17:24:00Z

+ const context: Record<string, unknown> = { Name: toPythonPackageName(flags["name"]) };
+
+ if (hasMetric) {
+ const raw = flags["metric"]!;


Can we use make this helper function and use zod here.

agentcore-devx-automation · 2026-08-31T17:28:09Z

Claude Security Review: no high-confidence findings. (run)

agentcore-devx-automation · 2026-08-31T17:31:40Z

Claude Security Review: no high-confidence findings. (run)

agentcore-devx-automation · 2026-08-31T17:35:47Z

Claude Security Review: no high-confidence findings. (run)

agentcore-devx-automation · 2026-08-31T17:53:03Z

Claude Security Review: no high-confidence findings. (run)

agentcore-devx-automation · 2026-08-31T18:20:07Z

Claude Security Review: no high-confidence findings. (run)

jariy17 · 2026-08-31T19:00:53Z

@@ -0,0 +1,15 @@
+{


I didn't want add an abstraction to generate common assets like this one due time constraints. We can look for this in the future.

agentcore-devx-automation · 2026-08-31T19:36:26Z

Claude Security Review: no high-confidence findings. (run)

Declarative code-based evaluators via projects. Mode is inferred from flags (mirrors CodeBasedConfigSchema managed XOR external): --lambda-arn -> external (BYO Lambda) --metric <library.Metric> -> managed 3P (deepeval/autoevals), scaffolded neither -> managed empty stub you fill in Scaffolds app/<name>/ from ported evaluator templates (python/deepeval/autoevals lambda), hardcodes codeLocation, and auto-wires additionalPolicies= [execution-role-policy.json]. Also enables `project remove evaluator`.

…aluators Runtimes, harnesses, and evaluators all scaffold into app/<name>, but the duplicate-name guard is per-resource-type and the tree write happens outside the rollback try/catch. An evaluator whose name matches an existing runtime/ harness dir (or a leftover from a removed evaluator) threw a raw 'File already exists' mid-write and orphaned partial files. Fail up front with a clear InputValidationError when app/<name> already exists.

…or code-based evaluators - Reject a namespaced/multi-dot metric class (e.g. deepeval.metrics.Faithfulness) that would render invalid Python; require a single class identifier. - --model is Bedrock-only: accept a bare model id / inference-profile-or- foundation-model ARN, optionally prefixed with bedrock/, validated via isValidBedrockModelId (same forms the llm-as-a-judge handler accepts). Non-Bedrock or slashless values now error instead of being silently dropped (deepeval) or passed to the wrong client (autoevals). - autoevals template prefixes bedrock/ for litellm routing now that Model is the bare id.

Print notes after add: the empty stub returns Pass for every session until implemented, and managed evaluators are scaffolded but not yet provisioned by 'project deploy' (no CDK/L3 support). External (--lambda-arn) prints neither.

…EOUT const; drop code comments

…templates layer Mirror the runtime layering: the handler now just parses/validates flags and passes a ManagedEvaluatorScaffoldInput; templates/evaluator.ts owns the library registry, per-library timeouts, render context, and buildManagedEvaluatorSpec (parallels buildRuntimeSpec). Also adds "evaluator" to RemoveResourceInput.

…aluators The L3 (@aws/agentcore-cdk) does provision spec.evaluators — synth emits AWS::BedrockAgentCore::Evaluator + Lambda — so the note was inaccurate. Keep the empty-stub 'returns Pass until implemented' note, which is still true.

agentcore-devx-automation · 2026-08-31T19:45:05Z

Claude Security Review: no high-confidence findings. (run)

jariy17 · 2026-08-28T22:14:37Z

What

Adds agentcore project add evaluator code-based — declarative code-based evaluators via projects — plus enables agentcore project remove evaluator. Follows the project add pattern (#2034 / #2004 / #2035 / #1998 / #2037) and, after review, mirrors the runtime layering (thin handler → templates/evaluator.ts owns the registry + buildManagedEvaluatorSpec, paralleling buildRuntimeSpec).

No mode/selector flag — the mode is inferred from what you pass (mirrors CodeBasedConfigSchema's managed XOR external):

You pass	Mode	Result
`--metric <library.Metric>`	managed · 3P	scaffolds a deepeval/autoevals Lambda from a template
(neither)	managed · empty	scaffolds an empty `@custom_code_based_evaluator()` stub you fill in
`--lambda-arn <arn>`	external	references an existing Lambda (no scaffold)

Command structure

agentcore project add evaluator add a custom evaluator to the current project
├── llm-as-a-judge existing — LLM prompted to score a session
└── code-based NEW — a Lambda that scores a session
agentcore project remove evaluator --name <name> NEW — enabled via the generic remove

agentcore project add evaluator code-based --help:

Usage: agentcore project add evaluator code-based [options]
add a code-based evaluator — a Lambda that scores a session. Pass a 3P metric,
an existing Lambda, or neither to scaffold an empty evaluator you fill in
Options:
--name <name> the name of the evaluator
--level <level> what to score: SESSION, TRACE, or TOOL_CALL
--metric <metric> 3P metric to scaffold as <library.Metric>,
e.g. deepeval.FaithfulnessMetric or autoevals.Factuality
--model <model> judge model for the 3P metric,
e.g. bedrock/anthropic.claude-3-5-sonnet-20240620-v1:0
--lambda-arn <lambda-arn> ARN of an existing Lambda that scores a session
--timeout-seconds <timeout-seconds> Lambda timeout in seconds (1-300)
--description <description> a description of what this evaluator measures
--kms-key-arn <kms-key-arn> customer-managed KMS key ARN to encrypt the evaluator
--tags <tags> tags to apply (JSON object of key/value strings)
-h, --help display help for command

Enforced in-handler (not shown by Commander): --name + --level required; exactly one of --metric/--lambda-arn; --metric must be <library>.<Class> where library ∈ {deepeval, autoevals} and Class is a single identifier; --model must be a Bedrock model id / inference-profile-or-foundation-model ARN (optionally bedrock/-prefixed) and requires --metric; --timeout-seconds/--model/--metric are managed-only. Managed auto-fills codeLocation=app/<name>, entrypoint=lambda_function.handler, per-library timeoutSeconds (deepeval 300, else 60), and additionalPolicies=["execution-role-policy.json"].

Commits

c4430c03 feat — the command + 3 scaffold templates + remove evaluator
baec1630 fix — guard app/<name> collisions (up-front, no partial writes)
e7bc3675 fix — validate --metric class + require a Bedrock --model
9bd79980 fix — echo the inferred mode + caveats at add time
32a10ef9 refactor — share toPythonPackageName via fsUtils; DEFAULT_TIMEOUT const
853dcf86 refactor — move template knowledge into templates/evaluator.ts (runtime layering)

Testing

bun run build OK · bun test src/handlers/project src/core/project → 597 pass / 0 fail.
Cloud bug bash (5 parallel agents; 2 deployed to a non-prod account, us-west-2, then tore down): all 5 flows + a 13-case error matrix pass at the CLI/scaffold/synth layer. The managed evaluator synthesizes correctly into AWS::BedrockAgentCore::Evaluator + Lambda + role + permissions, and generated Python ast.parses for deepeval + autoevals (bedrock + openai branches). Full report shared separately.

Known issues surfaced by the cloud deploy (both OUTSIDE this PR)

The evaluator authoring + synth work; end-to-end project deploy is currently blocked by two pre-existing bugs, neither in this feature's code:

Payments construct doesn't exist (feat(project): add payment resources #2120) — the vended src/assets/cdk/lib/cdk-stack.ts imports/instantiates AgentCorePayments, but no published @aws/agentcore-cdk version exports that name (alpha.49 and alpha.50 export AgentCorePaymentManager/Connector). So it's a template code bug, not a stale pin — a version bump can't fix it; the construct must be dropped or gated behind a spec payments-config check. tsc TS2305 → build/synth/deploy fail for every scaffolded project, evaluator or not. Owner: payments.
L3 evaluator "Access denied for Lambda" (reproducible) — with the payments line removed, synth emits the full evaluator resource set and deploy reaches AWS::BedrockAgentCore::Evaluator, which fails CREATE_FAILED: "Access denied for Lambda function …". Confirmed across two runs (not a race): the lambda:InvokeFunction/GetFunction grants to bedrock-agentcore.amazonaws.com reach CREATE_COMPLETE ~9s before the evaluator, yet the control-plane access check still denies — a deterministic authorization-shape mismatch (likely a required SourceAccount/SourceArn condition) in @aws/agentcore-cdk's AgentCoreEvaluator. Owner: agentcore-cdk L3.

🤖 Draft — CLI/authoring layer is complete and green; hold merge until the two deploy blockers (payments #2120, agentcore-cdk L3) land.

agentcore-devx-automation

AgentCore Harness Review

Verdict: Looks good

Small, focused fix that surfaces two real footguns of the code-based evaluator scaffold:

Empty stub silently returns Pass for every session (verified against src/assets/evaluators/python-lambda/lambda_function.py, which returns label="Pass").
Managed code-based evaluators aren't yet provisioned by project deploy.

Logic in index.ts (lines 158–166) matches the commit message: the "returns Pass" note is gated on !hasLambda && !hasMetric, and the "not yet provisioned" note is gated on !hasLambda, so --lambda-arn (external) correctly prints neither.

Tests in index.test.ts use real temp directories via mkdtemp and drive the handler through the router — no excessive mocking — and cover both the stub and external paths. Telemetry isn't warranted here since this only adds informational stderr output, not a new feature.

Nothing blocking.

codecov-commenter · 2026-08-28T22:18:03Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.31%. Comparing base (3d449c5) to head (9bd7998).

Additional details and impacted files

@@ Coverage Diff @@## refactor #2144 +/- ##
============================================
+ Coverage 97.29% 97.31% +0.01% 
============================================
Files 479 481 +2 Lines 29673 29865 +192 ============================================
+ Hits 28871 29063 +192 
Misses 802 802

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

agentcore-devx-automation · 2026-08-28T22:18:32Z

Claude Security Review: no high-confidence findings. (run)

agentcore-devx-automation · 2026-08-31T14:53:59Z

Claude Security Review: no high-confidence findings. (run)

jariy17 · 2026-08-31T17:24:00Z

+ const context: Record<string, unknown> = { Name: toPythonPackageName(flags["name"]) };
+
+ if (hasMetric) {
+ const raw = flags["metric"]!;


Can we use make this helper function and use zod here.

agentcore-devx-automation · 2026-08-31T17:28:09Z

Claude Security Review: no high-confidence findings. (run)

agentcore-devx-automation · 2026-08-31T17:31:40Z

Claude Security Review: no high-confidence findings. (run)

agentcore-devx-automation · 2026-08-31T17:35:47Z

Claude Security Review: no high-confidence findings. (run)

agentcore-devx-automation · 2026-08-31T17:53:03Z

Claude Security Review: no high-confidence findings. (run)

agentcore-devx-automation · 2026-08-31T18:20:07Z

Claude Security Review: no high-confidence findings. (run)

jariy17 · 2026-08-31T19:00:53Z

@@ -0,0 +1,15 @@
+{


I didn't want add an abstraction to generate common assets like this one due time constraints. We can look for this in the future.

agentcore-devx-automation · 2026-08-31T19:36:26Z

Claude Security Review: no high-confidence findings. (run)

Declarative code-based evaluators via projects. Mode is inferred from flags (mirrors CodeBasedConfigSchema managed XOR external): --lambda-arn -> external (BYO Lambda) --metric <library.Metric> -> managed 3P (deepeval/autoevals), scaffolded neither -> managed empty stub you fill in Scaffolds app/<name>/ from ported evaluator templates (python/deepeval/autoevals lambda), hardcodes codeLocation, and auto-wires additionalPolicies= [execution-role-policy.json]. Also enables `project remove evaluator`.

…aluators Runtimes, harnesses, and evaluators all scaffold into app/<name>, but the duplicate-name guard is per-resource-type and the tree write happens outside the rollback try/catch. An evaluator whose name matches an existing runtime/ harness dir (or a leftover from a removed evaluator) threw a raw 'File already exists' mid-write and orphaned partial files. Fail up front with a clear InputValidationError when app/<name> already exists.

…or code-based evaluators - Reject a namespaced/multi-dot metric class (e.g. deepeval.metrics.Faithfulness) that would render invalid Python; require a single class identifier. - --model is Bedrock-only: accept a bare model id / inference-profile-or- foundation-model ARN, optionally prefixed with bedrock/, validated via isValidBedrockModelId (same forms the llm-as-a-judge handler accepts). Non-Bedrock or slashless values now error instead of being silently dropped (deepeval) or passed to the wrong client (autoevals). - autoevals template prefixes bedrock/ for litellm routing now that Model is the bare id.

Print notes after add: the empty stub returns Pass for every session until implemented, and managed evaluators are scaffolded but not yet provisioned by 'project deploy' (no CDK/L3 support). External (--lambda-arn) prints neither.

…EOUT const; drop code comments

…templates layer Mirror the runtime layering: the handler now just parses/validates flags and passes a ManagedEvaluatorScaffoldInput; templates/evaluator.ts owns the library registry, per-library timeouts, render context, and buildManagedEvaluatorSpec (parallels buildRuntimeSpec). Also adds "evaluator" to RemoveResourceInput.

…aluators The L3 (@aws/agentcore-cdk) does provision spec.evaluators — synth emits AWS::BedrockAgentCore::Evaluator + Lambda — so the note was inaccurate. Keep the empty-stub 'returns Pass until implemented' note, which is still true.

agentcore-devx-automation · 2026-08-31T19:45:05Z

Claude Security Review: no high-confidence findings. (run)

jariy17 · 2026-08-28T22:14:37Z

What

Adds agentcore project add evaluator code-based — declarative code-based evaluators via projects — plus enables agentcore project remove evaluator. Follows the project add pattern (#2034 / #2004 / #2035 / #1998 / #2037) and, after review, mirrors the runtime layering (thin handler → templates/evaluator.ts owns the registry + buildManagedEvaluatorSpec, paralleling buildRuntimeSpec).

No mode/selector flag — the mode is inferred from what you pass (mirrors CodeBasedConfigSchema's managed XOR external):

You pass	Mode	Result
`--metric <library.Metric>`	managed · 3P	scaffolds a deepeval/autoevals Lambda from a template
(neither)	managed · empty	scaffolds an empty `@custom_code_based_evaluator()` stub you fill in
`--lambda-arn <arn>`	external	references an existing Lambda (no scaffold)

Command structure

agentcore project add evaluator add a custom evaluator to the current project
├── llm-as-a-judge existing — LLM prompted to score a session
└── code-based NEW — a Lambda that scores a session
agentcore project remove evaluator --name <name> NEW — enabled via the generic remove

agentcore project add evaluator code-based --help:

Usage: agentcore project add evaluator code-based [options]
add a code-based evaluator — a Lambda that scores a session. Pass a 3P metric,
an existing Lambda, or neither to scaffold an empty evaluator you fill in
Options:
--name <name> the name of the evaluator
--level <level> what to score: SESSION, TRACE, or TOOL_CALL
--metric <metric> 3P metric to scaffold as <library.Metric>,
e.g. deepeval.FaithfulnessMetric or autoevals.Factuality
--model <model> judge model for the 3P metric,
e.g. bedrock/anthropic.claude-3-5-sonnet-20240620-v1:0
--lambda-arn <lambda-arn> ARN of an existing Lambda that scores a session
--timeout-seconds <timeout-seconds> Lambda timeout in seconds (1-300)
--description <description> a description of what this evaluator measures
--kms-key-arn <kms-key-arn> customer-managed KMS key ARN to encrypt the evaluator
--tags <tags> tags to apply (JSON object of key/value strings)
-h, --help display help for command

Enforced in-handler (not shown by Commander): --name + --level required; exactly one of --metric/--lambda-arn; --metric must be <library>.<Class> where library ∈ {deepeval, autoevals} and Class is a single identifier; --model must be a Bedrock model id / inference-profile-or-foundation-model ARN (optionally bedrock/-prefixed) and requires --metric; --timeout-seconds/--model/--metric are managed-only. Managed auto-fills codeLocation=app/<name>, entrypoint=lambda_function.handler, per-library timeoutSeconds (deepeval 300, else 60), and additionalPolicies=["execution-role-policy.json"].

Commits

c4430c03 feat — the command + 3 scaffold templates + remove evaluator
baec1630 fix — guard app/<name> collisions (up-front, no partial writes)
e7bc3675 fix — validate --metric class + require a Bedrock --model
9bd79980 fix — echo the inferred mode + caveats at add time
32a10ef9 refactor — share toPythonPackageName via fsUtils; DEFAULT_TIMEOUT const
853dcf86 refactor — move template knowledge into templates/evaluator.ts (runtime layering)

Testing

bun run build OK · bun test src/handlers/project src/core/project → 597 pass / 0 fail.
Cloud bug bash (5 parallel agents; 2 deployed to a non-prod account, us-west-2, then tore down): all 5 flows + a 13-case error matrix pass at the CLI/scaffold/synth layer. The managed evaluator synthesizes correctly into AWS::BedrockAgentCore::Evaluator + Lambda + role + permissions, and generated Python ast.parses for deepeval + autoevals (bedrock + openai branches). Full report shared separately.

Known issues surfaced by the cloud deploy (both OUTSIDE this PR)

The evaluator authoring + synth work; end-to-end project deploy is currently blocked by two pre-existing bugs, neither in this feature's code:

Payments construct doesn't exist (feat(project): add payment resources #2120) — the vended src/assets/cdk/lib/cdk-stack.ts imports/instantiates AgentCorePayments, but no published @aws/agentcore-cdk version exports that name (alpha.49 and alpha.50 export AgentCorePaymentManager/Connector). So it's a template code bug, not a stale pin — a version bump can't fix it; the construct must be dropped or gated behind a spec payments-config check. tsc TS2305 → build/synth/deploy fail for every scaffolded project, evaluator or not. Owner: payments.
L3 evaluator "Access denied for Lambda" (reproducible) — with the payments line removed, synth emits the full evaluator resource set and deploy reaches AWS::BedrockAgentCore::Evaluator, which fails CREATE_FAILED: "Access denied for Lambda function …". Confirmed across two runs (not a race): the lambda:InvokeFunction/GetFunction grants to bedrock-agentcore.amazonaws.com reach CREATE_COMPLETE ~9s before the evaluator, yet the control-plane access check still denies — a deterministic authorization-shape mismatch (likely a required SourceAccount/SourceArn condition) in @aws/agentcore-cdk's AgentCoreEvaluator. Owner: agentcore-cdk L3.

🤖 Draft — CLI/authoring layer is complete and green; hold merge until the two deploy blockers (payments #2120, agentcore-cdk L3) land.

agentcore-devx-automation

AgentCore Harness Review

Verdict: Looks good

Small, focused fix that surfaces two real footguns of the code-based evaluator scaffold:

Empty stub silently returns Pass for every session (verified against src/assets/evaluators/python-lambda/lambda_function.py, which returns label="Pass").
Managed code-based evaluators aren't yet provisioned by project deploy.

Logic in index.ts (lines 158–166) matches the commit message: the "returns Pass" note is gated on !hasLambda && !hasMetric, and the "not yet provisioned" note is gated on !hasLambda, so --lambda-arn (external) correctly prints neither.

Tests in index.test.ts use real temp directories via mkdtemp and drive the handler through the router — no excessive mocking — and cover both the stub and external paths. Telemetry isn't warranted here since this only adds informational stderr output, not a new feature.

Nothing blocking.

codecov-commenter · 2026-08-28T22:18:03Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.31%. Comparing base (3d449c5) to head (9bd7998).

Additional details and impacted files

@@ Coverage Diff @@## refactor #2144 +/- ##
============================================
+ Coverage 97.29% 97.31% +0.01% 
============================================
Files 479 481 +2 Lines 29673 29865 +192 ============================================
+ Hits 28871 29063 +192 
Misses 802 802

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

agentcore-devx-automation · 2026-08-28T22:18:32Z

Claude Security Review: no high-confidence findings. (run)

agentcore-devx-automation · 2026-08-31T14:53:59Z

Claude Security Review: no high-confidence findings. (run)

jariy17 · 2026-08-31T17:24:00Z

+ const context: Record<string, unknown> = { Name: toPythonPackageName(flags["name"]) };
+
+ if (hasMetric) {
+ const raw = flags["metric"]!;


Can we use make this helper function and use zod here.

agentcore-devx-automation · 2026-08-31T17:28:09Z

Claude Security Review: no high-confidence findings. (run)

agentcore-devx-automation · 2026-08-31T17:31:40Z

Claude Security Review: no high-confidence findings. (run)

agentcore-devx-automation · 2026-08-31T17:35:47Z

Claude Security Review: no high-confidence findings. (run)

agentcore-devx-automation · 2026-08-31T17:53:03Z

Claude Security Review: no high-confidence findings. (run)

agentcore-devx-automation · 2026-08-31T18:20:07Z

Claude Security Review: no high-confidence findings. (run)

jariy17 · 2026-08-31T19:00:53Z

@@ -0,0 +1,15 @@
+{


I didn't want add an abstraction to generate common assets like this one due time constraints. We can look for this in the future.

agentcore-devx-automation · 2026-08-31T19:36:26Z

Claude Security Review: no high-confidence findings. (run)

Declarative code-based evaluators via projects. Mode is inferred from flags (mirrors CodeBasedConfigSchema managed XOR external): --lambda-arn -> external (BYO Lambda) --metric <library.Metric> -> managed 3P (deepeval/autoevals), scaffolded neither -> managed empty stub you fill in Scaffolds app/<name>/ from ported evaluator templates (python/deepeval/autoevals lambda), hardcodes codeLocation, and auto-wires additionalPolicies= [execution-role-policy.json]. Also enables `project remove evaluator`.

…aluators Runtimes, harnesses, and evaluators all scaffold into app/<name>, but the duplicate-name guard is per-resource-type and the tree write happens outside the rollback try/catch. An evaluator whose name matches an existing runtime/ harness dir (or a leftover from a removed evaluator) threw a raw 'File already exists' mid-write and orphaned partial files. Fail up front with a clear InputValidationError when app/<name> already exists.

…or code-based evaluators - Reject a namespaced/multi-dot metric class (e.g. deepeval.metrics.Faithfulness) that would render invalid Python; require a single class identifier. - --model is Bedrock-only: accept a bare model id / inference-profile-or- foundation-model ARN, optionally prefixed with bedrock/, validated via isValidBedrockModelId (same forms the llm-as-a-judge handler accepts). Non-Bedrock or slashless values now error instead of being silently dropped (deepeval) or passed to the wrong client (autoevals). - autoevals template prefixes bedrock/ for litellm routing now that Model is the bare id.

Print notes after add: the empty stub returns Pass for every session until implemented, and managed evaluators are scaffolded but not yet provisioned by 'project deploy' (no CDK/L3 support). External (--lambda-arn) prints neither.

…EOUT const; drop code comments

…templates layer Mirror the runtime layering: the handler now just parses/validates flags and passes a ManagedEvaluatorScaffoldInput; templates/evaluator.ts owns the library registry, per-library timeouts, render context, and buildManagedEvaluatorSpec (parallels buildRuntimeSpec). Also adds "evaluator" to RemoveResourceInput.

…aluators The L3 (@aws/agentcore-cdk) does provision spec.evaluators — synth emits AWS::BedrockAgentCore::Evaluator + Lambda — so the note was inaccurate. Keep the empty-stub 'returns Pass until implemented' note, which is still true.

agentcore-devx-automation · 2026-08-31T19:45:05Z

Claude Security Review: no high-confidence findings. (run)

github-actionsBot added the size/xl PR size: XL label Aug 28, 2026

agentcore-devx-automationBot added agentcore-harness-reviewing AgentCore Harness review in progress claude-security-reviewing Claude Code /security-review in progress labels Aug 28, 2026

agentcore-devx-automationBot reviewed Aug 28, 2026
View reviewed changes

agentcore-devx-automationBot removed the agentcore-harness-reviewing AgentCore Harness review in progress label Aug 28, 2026

agentcore-devx-automationBot removed the claude-security-reviewing Claude Code /security-review in progress label Aug 28, 2026

jariy17 force-pushed the feat/project-add-evaluator-code-based branch from 303c8db to 9bd7998Compare August 31, 2026 14:51

github-actionsBot added size/l PR size: L and removed size/xl PR size: XL labels Aug 31, 2026

agentcore-devx-automationBot added the claude-security-reviewing Claude Code /security-review in progress label Aug 31, 2026

agentcore-devx-automationBot removed the claude-security-reviewing Claude Code /security-review in progress label Aug 31, 2026

jariy17 commented Aug 31, 2026
View reviewed changes

github-actionsBot added size/l PR size: L and removed size/l PR size: L labels Aug 31, 2026

agentcore-devx-automationBot added the claude-security-reviewing Claude Code /security-review in progress label Aug 31, 2026

agentcore-devx-automationBot removed the claude-security-reviewing Claude Code /security-review in progress label Aug 31, 2026

github-actionsBot added size/l PR size: L and removed size/l PR size: L labels Aug 31, 2026

agentcore-devx-automationBot added the claude-security-reviewing Claude Code /security-review in progress label Aug 31, 2026

agentcore-devx-automationBot removed the claude-security-reviewing Claude Code /security-review in progress label Aug 31, 2026

jariy17 force-pushed the feat/project-add-evaluator-code-based branch from e78e9bc to 32a10efCompare August 31, 2026 17:33

github-actionsBot added size/l PR size: L and removed size/l PR size: L labels Aug 31, 2026

agentcore-devx-automationBot added the claude-security-reviewing Claude Code /security-review in progress label Aug 31, 2026

agentcore-devx-automationBot removed the claude-security-reviewing Claude Code /security-review in progress label Aug 31, 2026

github-actionsBot added size/l PR size: L and removed size/l PR size: L labels Aug 31, 2026

agentcore-devx-automationBot added the claude-security-reviewing Claude Code /security-review in progress label Aug 31, 2026

agentcore-devx-automationBot removed the claude-security-reviewing Claude Code /security-review in progress label Aug 31, 2026

github-actionsBot added size/l PR size: L and removed size/l PR size: L labels Aug 31, 2026

jariy17 commented Aug 31, 2026
View reviewed changes

github-actionsBot added size/l PR size: L and removed size/l PR size: L labels Aug 31, 2026

agentcore-devx-automationBot added the claude-security-reviewing Claude Code /security-review in progress label Aug 31, 2026

agentcore-devx-automationBot removed the claude-security-reviewing Claude Code /security-review in progress label Aug 31, 2026

jariy17 added 7 commits August 31, 2026 19:38

refactor(project): share toPythonPackageName via fsUtils; DEFAULT_TIM…
106aab0
…EOUT const; drop code comments

jariy17 force-pushed the feat/project-add-evaluator-code-based branch from e440cb5 to 7215450Compare August 31, 2026 19:41