feat(project): add online-insight to project add by jariy17 · Pull Request #2063 · aws/agentcore-cli

jariy17 · 2026-08-21T16:35:25Z

Command structure

Usage: agentcore project add online-insight [options]
adds an online insight config to the current project
Options:
--name <name> the name of the online insight config
--agent <agent> harness/runtime name whose traffic to sample (mutually exclusive with --log-group-name)
--endpoint <endpoint> the agent endpoint qualifier to scope monitoring to (requires --agent)
--log-group-name <name...> CloudWatch log group name(s) for custom data sources (1-5; mutually exclusive with --agent)
--service-name <name...> service name(s) to filter traces for custom data sources (requires --log-group-name)
--insight <insight...> insight ID(s) to apply: Builtin.Insight.* identifiers or full ARNs
--clustering-frequency <freq...> insight clustering cadence(s): DAILY, WEEKLY, MONTHLY
--sampling-rate <sampling-rate> percentage of sessions to sample (0.01-100)
--description <description> a description of the config's monitoring purpose
--enable-on-create <true|false> enable insights immediately after deploy (default true; pass false to add it paused)
--tags <tags> tags to apply (JSON object of key/value strings)

online-insight is the insights counterpart of online-eval (#2048): same schema, same onlineEvalConfigs array, same spec key — it fills insights instead of evaluators and adds --clustering-frequency. Source options are identical (--agent XOR --log-group-name).

agentcore.json fields written

Each invocation appends one entry to onlineEvalConfigs[] in agentcore/agentcore.json:

Field	From flag	Notes
`name`	`--name`	required; 1-48 chars, letter-led alnum/underscore
`agent`	`--agent`	runtime to monitor; XOR`logGroupNames`
`endpoint`	`--endpoint`	endpoint qualifier; requires `agent`
`logGroupNames`	`--log-group-name`	custom CloudWatch source (1-5); XOR`agent`
`serviceNames`	`--service-name`	trace filter; requires `logGroupNames`
`insights`	`--insight`	required; each `Builtin.Insight.*` or an ARN
`clusteringConfig.frequencies`	`--clustering-frequency`	subset of DAILY/WEEKLY/MONTHLY; requires insights
`samplingRate`	`--sampling-rate`	required; 0.01-100 (percent)
`description`	`--description`	<=200 chars
`enableOnCreate`	`--enable-on-create`	`true`/`false`; `false` deploys the config paused
`tags`	`--tags`	JSON key/value object

Cross-field rules are enforced by OnlineEvalConfigSchema (agent XOR log groups, endpoint→agent, serviceNames→log groups, clustering→insights). Insight-ID format (Builtin.Insight.* / ARN) is validated in the handler. --agent existence is checked project-wide on the post-write ProjectSpecSchema validation in addResource.

Implementation

Mirrors the merged online-eval PR — 1 new handler + 1 co-located test + 3 small wirings:

newsrc/handlers/project/add/online-insight/index.ts
newsrc/handlers/project/add/online-insight/index.test.ts
add/index.ts — register the handler
project/types.ts — add online-insight to AddResourceInput
core/project/manager.tsx — stack into the no-scaffold case + toProjectSpecKey

Verification

tsc --noEmit 0 errors; bun test — new suite 15/15, project+manager regression 148/148; prettier --check . clean.
Bug bash (22/22 pass): 9 happy + 11 error flows local, plus a real deploy to a sandbox account — synthesized AWS::BedrockAgentCore::OnlineEvaluationConfig (Insights wired, ENABLED at sampling 50), a second config deployed paused (--enable-on-create false → DISABLED), both verified via the control plane and torn down.

Related: aws/agentcore-l3-cdk-constructs#335 — OnlineEvaluationConfigagent resolves runtimes only (harness targets unsupported). The --agent help text here says "runtime" to match.

codecov-commenter · 2026-08-21T16:37:58Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.12%. Comparing base (71b0f0a) to head (3a1547e).
⚠️ Report is 1 commits behind head on refactor.

Additional details and impacted files

@@ Coverage Diff @@## refactor #2063 +/- ##
============================================
+ Coverage 97.10% 97.12% +0.01% 
============================================
Files 384 385 +1 Lines 22683 22790 +107 ============================================
+ Hits 22027 22134 +107 
Misses 656 656

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

nborges-aws · 2026-08-21T16:51:20Z

+ flag("name", "the name of the online insight config", z.string().optional()),
+ flag(
+ "agent",
+ "harness/runtime name whose traffic to sample (mutually exclusive with --log-group-name)",


The flag here says "harness/runtime", but looking at the projectSchema I'm pretty sure this will always fail for harness.
ProjectSpecSchema builds agentNames list from spec.runtimes. Harnesses are never read into this list. So when if (config.agent && !agentNames.has(config.agent)) runs, agentNames will never see harnesses and this will always fail.

Good catch — you're right. Properly supporting a harness target needs changes in the CDK constructs (a harness would have to resolve to its underlying runtime in both the validation set and the log-group derivation), so I've filed aws/agentcore-l3-cdk-constructs#335 to track that.
For now I've narrowed the --agent description to runtime-only so it stops promising harness support that isn't wired up. I've also opened a follow-up PR (#2065) to make the same one-line correction on online-eval.

nborges-aws

One comment to address about harness source for --agent failing validation. PR looks good otherwise

Only runtimes are valid online-eval targets end-to-end: the project superRefine builds agentNames from spec.runtimes, and the CDK construct resolves config.agent against the runtime-only environments map. The prior 'harness/runtime' wording promised harness support no layer implements. Addresses review comment.

tejaskash

LGTM

jariy17 · 2026-08-21T16:35:25Z

Command structure

Usage: agentcore project add online-insight [options]
adds an online insight config to the current project
Options:
--name <name> the name of the online insight config
--agent <agent> harness/runtime name whose traffic to sample (mutually exclusive with --log-group-name)
--endpoint <endpoint> the agent endpoint qualifier to scope monitoring to (requires --agent)
--log-group-name <name...> CloudWatch log group name(s) for custom data sources (1-5; mutually exclusive with --agent)
--service-name <name...> service name(s) to filter traces for custom data sources (requires --log-group-name)
--insight <insight...> insight ID(s) to apply: Builtin.Insight.* identifiers or full ARNs
--clustering-frequency <freq...> insight clustering cadence(s): DAILY, WEEKLY, MONTHLY
--sampling-rate <sampling-rate> percentage of sessions to sample (0.01-100)
--description <description> a description of the config's monitoring purpose
--enable-on-create <true|false> enable insights immediately after deploy (default true; pass false to add it paused)
--tags <tags> tags to apply (JSON object of key/value strings)

online-insight is the insights counterpart of online-eval (#2048): same schema, same onlineEvalConfigs array, same spec key — it fills insights instead of evaluators and adds --clustering-frequency. Source options are identical (--agent XOR --log-group-name).

agentcore.json fields written

Each invocation appends one entry to onlineEvalConfigs[] in agentcore/agentcore.json:

Field	From flag	Notes
`name`	`--name`	required; 1-48 chars, letter-led alnum/underscore
`agent`	`--agent`	runtime to monitor; XOR`logGroupNames`
`endpoint`	`--endpoint`	endpoint qualifier; requires `agent`
`logGroupNames`	`--log-group-name`	custom CloudWatch source (1-5); XOR`agent`
`serviceNames`	`--service-name`	trace filter; requires `logGroupNames`
`insights`	`--insight`	required; each `Builtin.Insight.*` or an ARN
`clusteringConfig.frequencies`	`--clustering-frequency`	subset of DAILY/WEEKLY/MONTHLY; requires insights
`samplingRate`	`--sampling-rate`	required; 0.01-100 (percent)
`description`	`--description`	<=200 chars
`enableOnCreate`	`--enable-on-create`	`true`/`false`; `false` deploys the config paused
`tags`	`--tags`	JSON key/value object

Cross-field rules are enforced by OnlineEvalConfigSchema (agent XOR log groups, endpoint→agent, serviceNames→log groups, clustering→insights). Insight-ID format (Builtin.Insight.* / ARN) is validated in the handler. --agent existence is checked project-wide on the post-write ProjectSpecSchema validation in addResource.

Implementation

Mirrors the merged online-eval PR — 1 new handler + 1 co-located test + 3 small wirings:

newsrc/handlers/project/add/online-insight/index.ts
newsrc/handlers/project/add/online-insight/index.test.ts
add/index.ts — register the handler
project/types.ts — add online-insight to AddResourceInput
core/project/manager.tsx — stack into the no-scaffold case + toProjectSpecKey

Verification

tsc --noEmit 0 errors; bun test — new suite 15/15, project+manager regression 148/148; prettier --check . clean.
Bug bash (22/22 pass): 9 happy + 11 error flows local, plus a real deploy to a sandbox account — synthesized AWS::BedrockAgentCore::OnlineEvaluationConfig (Insights wired, ENABLED at sampling 50), a second config deployed paused (--enable-on-create false → DISABLED), both verified via the control plane and torn down.

Related: aws/agentcore-l3-cdk-constructs#335 — OnlineEvaluationConfigagent resolves runtimes only (harness targets unsupported). The --agent help text here says "runtime" to match.

codecov-commenter · 2026-08-21T16:37:58Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.12%. Comparing base (71b0f0a) to head (3a1547e).
⚠️ Report is 1 commits behind head on refactor.

Additional details and impacted files

@@ Coverage Diff @@## refactor #2063 +/- ##
============================================
+ Coverage 97.10% 97.12% +0.01% 
============================================
Files 384 385 +1 Lines 22683 22790 +107 ============================================
+ Hits 22027 22134 +107 
Misses 656 656

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

nborges-aws · 2026-08-21T16:51:20Z

+ flag("name", "the name of the online insight config", z.string().optional()),
+ flag(
+ "agent",
+ "harness/runtime name whose traffic to sample (mutually exclusive with --log-group-name)",


The flag here says "harness/runtime", but looking at the projectSchema I'm pretty sure this will always fail for harness.
ProjectSpecSchema builds agentNames list from spec.runtimes. Harnesses are never read into this list. So when if (config.agent && !agentNames.has(config.agent)) runs, agentNames will never see harnesses and this will always fail.

Good catch — you're right. Properly supporting a harness target needs changes in the CDK constructs (a harness would have to resolve to its underlying runtime in both the validation set and the log-group derivation), so I've filed aws/agentcore-l3-cdk-constructs#335 to track that.
For now I've narrowed the --agent description to runtime-only so it stops promising harness support that isn't wired up. I've also opened a follow-up PR (#2065) to make the same one-line correction on online-eval.

nborges-aws

One comment to address about harness source for --agent failing validation. PR looks good otherwise

Only runtimes are valid online-eval targets end-to-end: the project superRefine builds agentNames from spec.runtimes, and the CDK construct resolves config.agent against the runtime-only environments map. The prior 'harness/runtime' wording promised harness support no layer implements. Addresses review comment.

tejaskash

LGTM

jariy17 · 2026-08-21T16:35:25Z

Command structure

Usage: agentcore project add online-insight [options]
adds an online insight config to the current project
Options:
--name <name> the name of the online insight config
--agent <agent> harness/runtime name whose traffic to sample (mutually exclusive with --log-group-name)
--endpoint <endpoint> the agent endpoint qualifier to scope monitoring to (requires --agent)
--log-group-name <name...> CloudWatch log group name(s) for custom data sources (1-5; mutually exclusive with --agent)
--service-name <name...> service name(s) to filter traces for custom data sources (requires --log-group-name)
--insight <insight...> insight ID(s) to apply: Builtin.Insight.* identifiers or full ARNs
--clustering-frequency <freq...> insight clustering cadence(s): DAILY, WEEKLY, MONTHLY
--sampling-rate <sampling-rate> percentage of sessions to sample (0.01-100)
--description <description> a description of the config's monitoring purpose
--enable-on-create <true|false> enable insights immediately after deploy (default true; pass false to add it paused)
--tags <tags> tags to apply (JSON object of key/value strings)

online-insight is the insights counterpart of online-eval (#2048): same schema, same onlineEvalConfigs array, same spec key — it fills insights instead of evaluators and adds --clustering-frequency. Source options are identical (--agent XOR --log-group-name).

agentcore.json fields written

Each invocation appends one entry to onlineEvalConfigs[] in agentcore/agentcore.json:

Field	From flag	Notes
`name`	`--name`	required; 1-48 chars, letter-led alnum/underscore
`agent`	`--agent`	runtime to monitor; XOR`logGroupNames`
`endpoint`	`--endpoint`	endpoint qualifier; requires `agent`
`logGroupNames`	`--log-group-name`	custom CloudWatch source (1-5); XOR`agent`
`serviceNames`	`--service-name`	trace filter; requires `logGroupNames`
`insights`	`--insight`	required; each `Builtin.Insight.*` or an ARN
`clusteringConfig.frequencies`	`--clustering-frequency`	subset of DAILY/WEEKLY/MONTHLY; requires insights
`samplingRate`	`--sampling-rate`	required; 0.01-100 (percent)
`description`	`--description`	<=200 chars
`enableOnCreate`	`--enable-on-create`	`true`/`false`; `false` deploys the config paused
`tags`	`--tags`	JSON key/value object

Cross-field rules are enforced by OnlineEvalConfigSchema (agent XOR log groups, endpoint→agent, serviceNames→log groups, clustering→insights). Insight-ID format (Builtin.Insight.* / ARN) is validated in the handler. --agent existence is checked project-wide on the post-write ProjectSpecSchema validation in addResource.

Implementation

Mirrors the merged online-eval PR — 1 new handler + 1 co-located test + 3 small wirings:

newsrc/handlers/project/add/online-insight/index.ts
newsrc/handlers/project/add/online-insight/index.test.ts
add/index.ts — register the handler
project/types.ts — add online-insight to AddResourceInput
core/project/manager.tsx — stack into the no-scaffold case + toProjectSpecKey

Verification

tsc --noEmit 0 errors; bun test — new suite 15/15, project+manager regression 148/148; prettier --check . clean.
Bug bash (22/22 pass): 9 happy + 11 error flows local, plus a real deploy to a sandbox account — synthesized AWS::BedrockAgentCore::OnlineEvaluationConfig (Insights wired, ENABLED at sampling 50), a second config deployed paused (--enable-on-create false → DISABLED), both verified via the control plane and torn down.

Related: aws/agentcore-l3-cdk-constructs#335 — OnlineEvaluationConfigagent resolves runtimes only (harness targets unsupported). The --agent help text here says "runtime" to match.

codecov-commenter · 2026-08-21T16:37:58Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.12%. Comparing base (71b0f0a) to head (3a1547e).
⚠️ Report is 1 commits behind head on refactor.

Additional details and impacted files

@@ Coverage Diff @@## refactor #2063 +/- ##
============================================
+ Coverage 97.10% 97.12% +0.01% 
============================================
Files 384 385 +1 Lines 22683 22790 +107 ============================================
+ Hits 22027 22134 +107 
Misses 656 656

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

nborges-aws · 2026-08-21T16:51:20Z

+ flag("name", "the name of the online insight config", z.string().optional()),
+ flag(
+ "agent",
+ "harness/runtime name whose traffic to sample (mutually exclusive with --log-group-name)",


The flag here says "harness/runtime", but looking at the projectSchema I'm pretty sure this will always fail for harness.
ProjectSpecSchema builds agentNames list from spec.runtimes. Harnesses are never read into this list. So when if (config.agent && !agentNames.has(config.agent)) runs, agentNames will never see harnesses and this will always fail.

Good catch — you're right. Properly supporting a harness target needs changes in the CDK constructs (a harness would have to resolve to its underlying runtime in both the validation set and the log-group derivation), so I've filed aws/agentcore-l3-cdk-constructs#335 to track that.
For now I've narrowed the --agent description to runtime-only so it stops promising harness support that isn't wired up. I've also opened a follow-up PR (#2065) to make the same one-line correction on online-eval.

nborges-aws

One comment to address about harness source for --agent failing validation. PR looks good otherwise

Only runtimes are valid online-eval targets end-to-end: the project superRefine builds agentNames from spec.runtimes, and the CDK construct resolves config.agent against the runtime-only environments map. The prior 'harness/runtime' wording promised harness support no layer implements. Addresses review comment.

tejaskash

LGTM

jariy17 · 2026-08-21T16:35:25Z

Command structure

Usage: agentcore project add online-insight [options]
adds an online insight config to the current project
Options:
--name <name> the name of the online insight config
--agent <agent> harness/runtime name whose traffic to sample (mutually exclusive with --log-group-name)
--endpoint <endpoint> the agent endpoint qualifier to scope monitoring to (requires --agent)
--log-group-name <name...> CloudWatch log group name(s) for custom data sources (1-5; mutually exclusive with --agent)
--service-name <name...> service name(s) to filter traces for custom data sources (requires --log-group-name)
--insight <insight...> insight ID(s) to apply: Builtin.Insight.* identifiers or full ARNs
--clustering-frequency <freq...> insight clustering cadence(s): DAILY, WEEKLY, MONTHLY
--sampling-rate <sampling-rate> percentage of sessions to sample (0.01-100)
--description <description> a description of the config's monitoring purpose
--enable-on-create <true|false> enable insights immediately after deploy (default true; pass false to add it paused)
--tags <tags> tags to apply (JSON object of key/value strings)

online-insight is the insights counterpart of online-eval (#2048): same schema, same onlineEvalConfigs array, same spec key — it fills insights instead of evaluators and adds --clustering-frequency. Source options are identical (--agent XOR --log-group-name).

agentcore.json fields written

Each invocation appends one entry to onlineEvalConfigs[] in agentcore/agentcore.json:

Field	From flag	Notes
`name`	`--name`	required; 1-48 chars, letter-led alnum/underscore
`agent`	`--agent`	runtime to monitor; XOR`logGroupNames`
`endpoint`	`--endpoint`	endpoint qualifier; requires `agent`
`logGroupNames`	`--log-group-name`	custom CloudWatch source (1-5); XOR`agent`
`serviceNames`	`--service-name`	trace filter; requires `logGroupNames`
`insights`	`--insight`	required; each `Builtin.Insight.*` or an ARN
`clusteringConfig.frequencies`	`--clustering-frequency`	subset of DAILY/WEEKLY/MONTHLY; requires insights
`samplingRate`	`--sampling-rate`	required; 0.01-100 (percent)
`description`	`--description`	<=200 chars
`enableOnCreate`	`--enable-on-create`	`true`/`false`; `false` deploys the config paused
`tags`	`--tags`	JSON key/value object

Cross-field rules are enforced by OnlineEvalConfigSchema (agent XOR log groups, endpoint→agent, serviceNames→log groups, clustering→insights). Insight-ID format (Builtin.Insight.* / ARN) is validated in the handler. --agent existence is checked project-wide on the post-write ProjectSpecSchema validation in addResource.

Implementation

Mirrors the merged online-eval PR — 1 new handler + 1 co-located test + 3 small wirings:

newsrc/handlers/project/add/online-insight/index.ts
newsrc/handlers/project/add/online-insight/index.test.ts
add/index.ts — register the handler
project/types.ts — add online-insight to AddResourceInput
core/project/manager.tsx — stack into the no-scaffold case + toProjectSpecKey

Verification

tsc --noEmit 0 errors; bun test — new suite 15/15, project+manager regression 148/148; prettier --check . clean.
Bug bash (22/22 pass): 9 happy + 11 error flows local, plus a real deploy to a sandbox account — synthesized AWS::BedrockAgentCore::OnlineEvaluationConfig (Insights wired, ENABLED at sampling 50), a second config deployed paused (--enable-on-create false → DISABLED), both verified via the control plane and torn down.

Related: aws/agentcore-l3-cdk-constructs#335 — OnlineEvaluationConfigagent resolves runtimes only (harness targets unsupported). The --agent help text here says "runtime" to match.

codecov-commenter · 2026-08-21T16:37:58Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.12%. Comparing base (71b0f0a) to head (3a1547e).
⚠️ Report is 1 commits behind head on refactor.

Additional details and impacted files

@@ Coverage Diff @@## refactor #2063 +/- ##
============================================
+ Coverage 97.10% 97.12% +0.01% 
============================================
Files 384 385 +1 Lines 22683 22790 +107 ============================================
+ Hits 22027 22134 +107 
Misses 656 656

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

nborges-aws · 2026-08-21T16:51:20Z

+ flag("name", "the name of the online insight config", z.string().optional()),
+ flag(
+ "agent",
+ "harness/runtime name whose traffic to sample (mutually exclusive with --log-group-name)",


The flag here says "harness/runtime", but looking at the projectSchema I'm pretty sure this will always fail for harness.
ProjectSpecSchema builds agentNames list from spec.runtimes. Harnesses are never read into this list. So when if (config.agent && !agentNames.has(config.agent)) runs, agentNames will never see harnesses and this will always fail.

Good catch — you're right. Properly supporting a harness target needs changes in the CDK constructs (a harness would have to resolve to its underlying runtime in both the validation set and the log-group derivation), so I've filed aws/agentcore-l3-cdk-constructs#335 to track that.
For now I've narrowed the --agent description to runtime-only so it stops promising harness support that isn't wired up. I've also opened a follow-up PR (#2065) to make the same one-line correction on online-eval.

nborges-aws

One comment to address about harness source for --agent failing validation. PR looks good otherwise

Only runtimes are valid online-eval targets end-to-end: the project superRefine builds agentNames from spec.runtimes, and the CDK construct resolves config.agent against the runtime-only environments map. The prior 'harness/runtime' wording promised harness support no layer implements. Addresses review comment.

tejaskash

LGTM

jariy17 · 2026-08-21T16:35:25Z

Command structure

Usage: agentcore project add online-insight [options]
adds an online insight config to the current project
Options:
--name <name> the name of the online insight config
--agent <agent> harness/runtime name whose traffic to sample (mutually exclusive with --log-group-name)
--endpoint <endpoint> the agent endpoint qualifier to scope monitoring to (requires --agent)
--log-group-name <name...> CloudWatch log group name(s) for custom data sources (1-5; mutually exclusive with --agent)
--service-name <name...> service name(s) to filter traces for custom data sources (requires --log-group-name)
--insight <insight...> insight ID(s) to apply: Builtin.Insight.* identifiers or full ARNs
--clustering-frequency <freq...> insight clustering cadence(s): DAILY, WEEKLY, MONTHLY
--sampling-rate <sampling-rate> percentage of sessions to sample (0.01-100)
--description <description> a description of the config's monitoring purpose
--enable-on-create <true|false> enable insights immediately after deploy (default true; pass false to add it paused)
--tags <tags> tags to apply (JSON object of key/value strings)

online-insight is the insights counterpart of online-eval (#2048): same schema, same onlineEvalConfigs array, same spec key — it fills insights instead of evaluators and adds --clustering-frequency. Source options are identical (--agent XOR --log-group-name).

agentcore.json fields written

Each invocation appends one entry to onlineEvalConfigs[] in agentcore/agentcore.json:

Field	From flag	Notes
`name`	`--name`	required; 1-48 chars, letter-led alnum/underscore
`agent`	`--agent`	runtime to monitor; XOR`logGroupNames`
`endpoint`	`--endpoint`	endpoint qualifier; requires `agent`
`logGroupNames`	`--log-group-name`	custom CloudWatch source (1-5); XOR`agent`
`serviceNames`	`--service-name`	trace filter; requires `logGroupNames`
`insights`	`--insight`	required; each `Builtin.Insight.*` or an ARN
`clusteringConfig.frequencies`	`--clustering-frequency`	subset of DAILY/WEEKLY/MONTHLY; requires insights
`samplingRate`	`--sampling-rate`	required; 0.01-100 (percent)
`description`	`--description`	<=200 chars
`enableOnCreate`	`--enable-on-create`	`true`/`false`; `false` deploys the config paused
`tags`	`--tags`	JSON key/value object

Cross-field rules are enforced by OnlineEvalConfigSchema (agent XOR log groups, endpoint→agent, serviceNames→log groups, clustering→insights). Insight-ID format (Builtin.Insight.* / ARN) is validated in the handler. --agent existence is checked project-wide on the post-write ProjectSpecSchema validation in addResource.

Implementation

Mirrors the merged online-eval PR — 1 new handler + 1 co-located test + 3 small wirings:

newsrc/handlers/project/add/online-insight/index.ts
newsrc/handlers/project/add/online-insight/index.test.ts
add/index.ts — register the handler
project/types.ts — add online-insight to AddResourceInput
core/project/manager.tsx — stack into the no-scaffold case + toProjectSpecKey

Verification

tsc --noEmit 0 errors; bun test — new suite 15/15, project+manager regression 148/148; prettier --check . clean.
Bug bash (22/22 pass): 9 happy + 11 error flows local, plus a real deploy to a sandbox account — synthesized AWS::BedrockAgentCore::OnlineEvaluationConfig (Insights wired, ENABLED at sampling 50), a second config deployed paused (--enable-on-create false → DISABLED), both verified via the control plane and torn down.

Related: aws/agentcore-l3-cdk-constructs#335 — OnlineEvaluationConfigagent resolves runtimes only (harness targets unsupported). The --agent help text here says "runtime" to match.

codecov-commenter · 2026-08-21T16:37:58Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.12%. Comparing base (71b0f0a) to head (3a1547e).
⚠️ Report is 1 commits behind head on refactor.

Additional details and impacted files

@@ Coverage Diff @@## refactor #2063 +/- ##
============================================
+ Coverage 97.10% 97.12% +0.01% 
============================================
Files 384 385 +1 Lines 22683 22790 +107 ============================================
+ Hits 22027 22134 +107 
Misses 656 656

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

nborges-aws · 2026-08-21T16:51:20Z

+ flag("name", "the name of the online insight config", z.string().optional()),
+ flag(
+ "agent",
+ "harness/runtime name whose traffic to sample (mutually exclusive with --log-group-name)",


The flag here says "harness/runtime", but looking at the projectSchema I'm pretty sure this will always fail for harness.
ProjectSpecSchema builds agentNames list from spec.runtimes. Harnesses are never read into this list. So when if (config.agent && !agentNames.has(config.agent)) runs, agentNames will never see harnesses and this will always fail.

Good catch — you're right. Properly supporting a harness target needs changes in the CDK constructs (a harness would have to resolve to its underlying runtime in both the validation set and the log-group derivation), so I've filed aws/agentcore-l3-cdk-constructs#335 to track that.
For now I've narrowed the --agent description to runtime-only so it stops promising harness support that isn't wired up. I've also opened a follow-up PR (#2065) to make the same one-line correction on online-eval.

nborges-aws

One comment to address about harness source for --agent failing validation. PR looks good otherwise

Only runtimes are valid online-eval targets end-to-end: the project superRefine builds agentNames from spec.runtimes, and the CDK construct resolves config.agent against the runtime-only environments map. The prior 'harness/runtime' wording promised harness support no layer implements. Addresses review comment.

tejaskash

LGTM

jariy17 · 2026-08-21T16:35:25Z

Command structure

Usage: agentcore project add online-insight [options]
adds an online insight config to the current project
Options:
--name <name> the name of the online insight config
--agent <agent> harness/runtime name whose traffic to sample (mutually exclusive with --log-group-name)
--endpoint <endpoint> the agent endpoint qualifier to scope monitoring to (requires --agent)
--log-group-name <name...> CloudWatch log group name(s) for custom data sources (1-5; mutually exclusive with --agent)
--service-name <name...> service name(s) to filter traces for custom data sources (requires --log-group-name)
--insight <insight...> insight ID(s) to apply: Builtin.Insight.* identifiers or full ARNs
--clustering-frequency <freq...> insight clustering cadence(s): DAILY, WEEKLY, MONTHLY
--sampling-rate <sampling-rate> percentage of sessions to sample (0.01-100)
--description <description> a description of the config's monitoring purpose
--enable-on-create <true|false> enable insights immediately after deploy (default true; pass false to add it paused)
--tags <tags> tags to apply (JSON object of key/value strings)

online-insight is the insights counterpart of online-eval (#2048): same schema, same onlineEvalConfigs array, same spec key — it fills insights instead of evaluators and adds --clustering-frequency. Source options are identical (--agent XOR --log-group-name).

agentcore.json fields written

Each invocation appends one entry to onlineEvalConfigs[] in agentcore/agentcore.json:

Field	From flag	Notes
`name`	`--name`	required; 1-48 chars, letter-led alnum/underscore
`agent`	`--agent`	runtime to monitor; XOR`logGroupNames`
`endpoint`	`--endpoint`	endpoint qualifier; requires `agent`
`logGroupNames`	`--log-group-name`	custom CloudWatch source (1-5); XOR`agent`
`serviceNames`	`--service-name`	trace filter; requires `logGroupNames`
`insights`	`--insight`	required; each `Builtin.Insight.*` or an ARN
`clusteringConfig.frequencies`	`--clustering-frequency`	subset of DAILY/WEEKLY/MONTHLY; requires insights
`samplingRate`	`--sampling-rate`	required; 0.01-100 (percent)
`description`	`--description`	<=200 chars
`enableOnCreate`	`--enable-on-create`	`true`/`false`; `false` deploys the config paused
`tags`	`--tags`	JSON key/value object

Cross-field rules are enforced by OnlineEvalConfigSchema (agent XOR log groups, endpoint→agent, serviceNames→log groups, clustering→insights). Insight-ID format (Builtin.Insight.* / ARN) is validated in the handler. --agent existence is checked project-wide on the post-write ProjectSpecSchema validation in addResource.

Implementation

Mirrors the merged online-eval PR — 1 new handler + 1 co-located test + 3 small wirings:

newsrc/handlers/project/add/online-insight/index.ts
newsrc/handlers/project/add/online-insight/index.test.ts
add/index.ts — register the handler
project/types.ts — add online-insight to AddResourceInput
core/project/manager.tsx — stack into the no-scaffold case + toProjectSpecKey

Verification

tsc --noEmit 0 errors; bun test — new suite 15/15, project+manager regression 148/148; prettier --check . clean.
Bug bash (22/22 pass): 9 happy + 11 error flows local, plus a real deploy to a sandbox account — synthesized AWS::BedrockAgentCore::OnlineEvaluationConfig (Insights wired, ENABLED at sampling 50), a second config deployed paused (--enable-on-create false → DISABLED), both verified via the control plane and torn down.

Related: aws/agentcore-l3-cdk-constructs#335 — OnlineEvaluationConfigagent resolves runtimes only (harness targets unsupported). The --agent help text here says "runtime" to match.

codecov-commenter · 2026-08-21T16:37:58Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.12%. Comparing base (71b0f0a) to head (3a1547e).
⚠️ Report is 1 commits behind head on refactor.

Additional details and impacted files

@@ Coverage Diff @@## refactor #2063 +/- ##
============================================
+ Coverage 97.10% 97.12% +0.01% 
============================================
Files 384 385 +1 Lines 22683 22790 +107 ============================================
+ Hits 22027 22134 +107 
Misses 656 656

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

nborges-aws · 2026-08-21T16:51:20Z

+ flag("name", "the name of the online insight config", z.string().optional()),
+ flag(
+ "agent",
+ "harness/runtime name whose traffic to sample (mutually exclusive with --log-group-name)",


The flag here says "harness/runtime", but looking at the projectSchema I'm pretty sure this will always fail for harness.
ProjectSpecSchema builds agentNames list from spec.runtimes. Harnesses are never read into this list. So when if (config.agent && !agentNames.has(config.agent)) runs, agentNames will never see harnesses and this will always fail.

Good catch — you're right. Properly supporting a harness target needs changes in the CDK constructs (a harness would have to resolve to its underlying runtime in both the validation set and the log-group derivation), so I've filed aws/agentcore-l3-cdk-constructs#335 to track that.
For now I've narrowed the --agent description to runtime-only so it stops promising harness support that isn't wired up. I've also opened a follow-up PR (#2065) to make the same one-line correction on online-eval.

nborges-aws

One comment to address about harness source for --agent failing validation. PR looks good otherwise

Only runtimes are valid online-eval targets end-to-end: the project superRefine builds agentNames from spec.runtimes, and the CDK construct resolves config.agent against the runtime-only environments map. The prior 'harness/runtime' wording promised harness support no layer implements. Addresses review comment.

tejaskash

LGTM

jariy17 · 2026-08-21T16:35:25Z

Command structure

Usage: agentcore project add online-insight [options]
adds an online insight config to the current project
Options:
--name <name> the name of the online insight config
--agent <agent> harness/runtime name whose traffic to sample (mutually exclusive with --log-group-name)
--endpoint <endpoint> the agent endpoint qualifier to scope monitoring to (requires --agent)
--log-group-name <name...> CloudWatch log group name(s) for custom data sources (1-5; mutually exclusive with --agent)
--service-name <name...> service name(s) to filter traces for custom data sources (requires --log-group-name)
--insight <insight...> insight ID(s) to apply: Builtin.Insight.* identifiers or full ARNs
--clustering-frequency <freq...> insight clustering cadence(s): DAILY, WEEKLY, MONTHLY
--sampling-rate <sampling-rate> percentage of sessions to sample (0.01-100)
--description <description> a description of the config's monitoring purpose
--enable-on-create <true|false> enable insights immediately after deploy (default true; pass false to add it paused)
--tags <tags> tags to apply (JSON object of key/value strings)

online-insight is the insights counterpart of online-eval (#2048): same schema, same onlineEvalConfigs array, same spec key — it fills insights instead of evaluators and adds --clustering-frequency. Source options are identical (--agent XOR --log-group-name).

agentcore.json fields written

Each invocation appends one entry to onlineEvalConfigs[] in agentcore/agentcore.json:

Field	From flag	Notes
`name`	`--name`	required; 1-48 chars, letter-led alnum/underscore
`agent`	`--agent`	runtime to monitor; XOR`logGroupNames`
`endpoint`	`--endpoint`	endpoint qualifier; requires `agent`
`logGroupNames`	`--log-group-name`	custom CloudWatch source (1-5); XOR`agent`
`serviceNames`	`--service-name`	trace filter; requires `logGroupNames`
`insights`	`--insight`	required; each `Builtin.Insight.*` or an ARN
`clusteringConfig.frequencies`	`--clustering-frequency`	subset of DAILY/WEEKLY/MONTHLY; requires insights
`samplingRate`	`--sampling-rate`	required; 0.01-100 (percent)
`description`	`--description`	<=200 chars
`enableOnCreate`	`--enable-on-create`	`true`/`false`; `false` deploys the config paused
`tags`	`--tags`	JSON key/value object

Cross-field rules are enforced by OnlineEvalConfigSchema (agent XOR log groups, endpoint→agent, serviceNames→log groups, clustering→insights). Insight-ID format (Builtin.Insight.* / ARN) is validated in the handler. --agent existence is checked project-wide on the post-write ProjectSpecSchema validation in addResource.

Implementation

Mirrors the merged online-eval PR — 1 new handler + 1 co-located test + 3 small wirings:

newsrc/handlers/project/add/online-insight/index.ts
newsrc/handlers/project/add/online-insight/index.test.ts
add/index.ts — register the handler
project/types.ts — add online-insight to AddResourceInput
core/project/manager.tsx — stack into the no-scaffold case + toProjectSpecKey

Verification

tsc --noEmit 0 errors; bun test — new suite 15/15, project+manager regression 148/148; prettier --check . clean.
Bug bash (22/22 pass): 9 happy + 11 error flows local, plus a real deploy to a sandbox account — synthesized AWS::BedrockAgentCore::OnlineEvaluationConfig (Insights wired, ENABLED at sampling 50), a second config deployed paused (--enable-on-create false → DISABLED), both verified via the control plane and torn down.

Related: aws/agentcore-l3-cdk-constructs#335 — OnlineEvaluationConfigagent resolves runtimes only (harness targets unsupported). The --agent help text here says "runtime" to match.

codecov-commenter · 2026-08-21T16:37:58Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.12%. Comparing base (71b0f0a) to head (3a1547e).
⚠️ Report is 1 commits behind head on refactor.

Additional details and impacted files

@@ Coverage Diff @@## refactor #2063 +/- ##
============================================
+ Coverage 97.10% 97.12% +0.01% 
============================================
Files 384 385 +1 Lines 22683 22790 +107 ============================================
+ Hits 22027 22134 +107 
Misses 656 656

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

nborges-aws · 2026-08-21T16:51:20Z

+ flag("name", "the name of the online insight config", z.string().optional()),
+ flag(
+ "agent",
+ "harness/runtime name whose traffic to sample (mutually exclusive with --log-group-name)",


The flag here says "harness/runtime", but looking at the projectSchema I'm pretty sure this will always fail for harness.
ProjectSpecSchema builds agentNames list from spec.runtimes. Harnesses are never read into this list. So when if (config.agent && !agentNames.has(config.agent)) runs, agentNames will never see harnesses and this will always fail.

Good catch — you're right. Properly supporting a harness target needs changes in the CDK constructs (a harness would have to resolve to its underlying runtime in both the validation set and the log-group derivation), so I've filed aws/agentcore-l3-cdk-constructs#335 to track that.
For now I've narrowed the --agent description to runtime-only so it stops promising harness support that isn't wired up. I've also opened a follow-up PR (#2065) to make the same one-line correction on online-eval.

nborges-aws

One comment to address about harness source for --agent failing validation. PR looks good otherwise

Only runtimes are valid online-eval targets end-to-end: the project superRefine builds agentNames from spec.runtimes, and the CDK construct resolves config.agent against the runtime-only environments map. The prior 'harness/runtime' wording promised harness support no layer implements. Addresses review comment.

tejaskash

LGTM

jariy17 · 2026-08-21T16:35:25Z

Command structure

Usage: agentcore project add online-insight [options]
adds an online insight config to the current project
Options:
--name <name> the name of the online insight config
--agent <agent> harness/runtime name whose traffic to sample (mutually exclusive with --log-group-name)
--endpoint <endpoint> the agent endpoint qualifier to scope monitoring to (requires --agent)
--log-group-name <name...> CloudWatch log group name(s) for custom data sources (1-5; mutually exclusive with --agent)
--service-name <name...> service name(s) to filter traces for custom data sources (requires --log-group-name)
--insight <insight...> insight ID(s) to apply: Builtin.Insight.* identifiers or full ARNs
--clustering-frequency <freq...> insight clustering cadence(s): DAILY, WEEKLY, MONTHLY
--sampling-rate <sampling-rate> percentage of sessions to sample (0.01-100)
--description <description> a description of the config's monitoring purpose
--enable-on-create <true|false> enable insights immediately after deploy (default true; pass false to add it paused)
--tags <tags> tags to apply (JSON object of key/value strings)

online-insight is the insights counterpart of online-eval (#2048): same schema, same onlineEvalConfigs array, same spec key — it fills insights instead of evaluators and adds --clustering-frequency. Source options are identical (--agent XOR --log-group-name).

agentcore.json fields written

Each invocation appends one entry to onlineEvalConfigs[] in agentcore/agentcore.json:

Field	From flag	Notes
`name`	`--name`	required; 1-48 chars, letter-led alnum/underscore
`agent`	`--agent`	runtime to monitor; XOR`logGroupNames`
`endpoint`	`--endpoint`	endpoint qualifier; requires `agent`
`logGroupNames`	`--log-group-name`	custom CloudWatch source (1-5); XOR`agent`
`serviceNames`	`--service-name`	trace filter; requires `logGroupNames`
`insights`	`--insight`	required; each `Builtin.Insight.*` or an ARN
`clusteringConfig.frequencies`	`--clustering-frequency`	subset of DAILY/WEEKLY/MONTHLY; requires insights
`samplingRate`	`--sampling-rate`	required; 0.01-100 (percent)
`description`	`--description`	<=200 chars
`enableOnCreate`	`--enable-on-create`	`true`/`false`; `false` deploys the config paused
`tags`	`--tags`	JSON key/value object

Cross-field rules are enforced by OnlineEvalConfigSchema (agent XOR log groups, endpoint→agent, serviceNames→log groups, clustering→insights). Insight-ID format (Builtin.Insight.* / ARN) is validated in the handler. --agent existence is checked project-wide on the post-write ProjectSpecSchema validation in addResource.

Implementation

Mirrors the merged online-eval PR — 1 new handler + 1 co-located test + 3 small wirings:

newsrc/handlers/project/add/online-insight/index.ts
newsrc/handlers/project/add/online-insight/index.test.ts
add/index.ts — register the handler
project/types.ts — add online-insight to AddResourceInput
core/project/manager.tsx — stack into the no-scaffold case + toProjectSpecKey

Verification

tsc --noEmit 0 errors; bun test — new suite 15/15, project+manager regression 148/148; prettier --check . clean.
Bug bash (22/22 pass): 9 happy + 11 error flows local, plus a real deploy to a sandbox account — synthesized AWS::BedrockAgentCore::OnlineEvaluationConfig (Insights wired, ENABLED at sampling 50), a second config deployed paused (--enable-on-create false → DISABLED), both verified via the control plane and torn down.

Related: aws/agentcore-l3-cdk-constructs#335 — OnlineEvaluationConfigagent resolves runtimes only (harness targets unsupported). The --agent help text here says "runtime" to match.

codecov-commenter · 2026-08-21T16:37:58Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.12%. Comparing base (71b0f0a) to head (3a1547e).
⚠️ Report is 1 commits behind head on refactor.

Additional details and impacted files

@@ Coverage Diff @@## refactor #2063 +/- ##
============================================
+ Coverage 97.10% 97.12% +0.01% 
============================================
Files 384 385 +1 Lines 22683 22790 +107 ============================================
+ Hits 22027 22134 +107 
Misses 656 656

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

nborges-aws · 2026-08-21T16:51:20Z

+ flag("name", "the name of the online insight config", z.string().optional()),
+ flag(
+ "agent",
+ "harness/runtime name whose traffic to sample (mutually exclusive with --log-group-name)",


The flag here says "harness/runtime", but looking at the projectSchema I'm pretty sure this will always fail for harness.
ProjectSpecSchema builds agentNames list from spec.runtimes. Harnesses are never read into this list. So when if (config.agent && !agentNames.has(config.agent)) runs, agentNames will never see harnesses and this will always fail.

Good catch — you're right. Properly supporting a harness target needs changes in the CDK constructs (a harness would have to resolve to its underlying runtime in both the validation set and the log-group derivation), so I've filed aws/agentcore-l3-cdk-constructs#335 to track that.
For now I've narrowed the --agent description to runtime-only so it stops promising harness support that isn't wired up. I've also opened a follow-up PR (#2065) to make the same one-line correction on online-eval.

nborges-aws

One comment to address about harness source for --agent failing validation. PR looks good otherwise

Only runtimes are valid online-eval targets end-to-end: the project superRefine builds agentNames from spec.runtimes, and the CDK construct resolves config.agent against the runtime-only environments map. The prior 'harness/runtime' wording promised harness support no layer implements. Addresses review comment.

tejaskash

LGTM

feat(project): add online-insight to project add
83b7eca

github-actionsBot added the agentcore-harness-reviewing AgentCore Harness review in progress label Aug 21, 2026

github-actionsBot removed the agentcore-harness-reviewing AgentCore Harness review in progress label Aug 21, 2026

nborges-aws reviewed Aug 21, 2026
View reviewed changes

jariy17 mentioned this pull request Aug 21, 2026
fix(online-eval): correct --agent help to runtime-only #2065
Merged

nborges-aws approved these changes Aug 21, 2026
View reviewed changes

tejaskash approved these changes Aug 21, 2026
View reviewed changes

tejaskash merged commit a41b4de into refactorAug 21, 2026
12 checks passed

tejaskash deleted the feat/add-project-online-insight branch August 21, 2026 18:50