refactor(eval): address on-demand review feedback by jariy17 · Pull Request #2007 · aws/agentcore-cli

jariy17 · 2026-08-14T20:19:03Z

Follow-up to #1983: makes on-demand evaluation failures easier to classify and removes review-identified noise.

Changes

Missing runtime telemetry now throws ResourceNotFoundError.
Failed CloudWatch queries now throw CloudWatchQueryError with query status and ID metadata.
Malformed telemetry produces at most one warning per command.
Repeated validation tests now use test.each.
Redundant and drift-prone comments were removed.

Decisions

Fail fast: one session can require multiple Evaluate calls. Continuing after one fails could return incomplete results while appearing successful.
Keep results flat:context.spanContext.sessionId supports per-session filtering without changing the SDK response shape.
Defer a generic poller: extract one when another production caller needs the same behavior.
Keep window resolvers separate: batch feeds the API's DataSourceConfig, including CloudWatch log groups and onlineEvaluationConfigArn; on-demand feeds client-side Logs Insights.

Verification

Build, typecheck, lint, changed-file formatting, and focused tests pass.
Full suite: 1,373 tests pass.

codecov-commenter · 2026-08-14T20:21:27Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.13%. Comparing base (2e18071) to head (8ffdcb9).

Additional details and impacted files

@@ Coverage Diff @@## refactor #2007 +/- ##
============================================
+ Coverage 97.09% 97.13% +0.04% 
============================================
Files 381 381 Lines 22778 22786 +8 ============================================
+ Hits 22116 22134 +18 + Misses 662 652 -10

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Hweinstock

thanks for coming back to this! one question on the testing pattern, otherwise lgtm.

Hweinstock · 2026-08-15T03:05:51Z

 return { core, stdout: io.stdout(), stderr: io.stderr() };
 }

+async function runWithRealCore(options: LogsOptions, logger = createSilentLogger()) {


should this use the fixture pattern? Might be a little harder to trigger these cases, but seems possible unless i'm missing something.

Kept it on inline stubs on purpose. The fixture harness records against a live account; these cases can't be provoked there:
Query resolving Failed/Cancelled/Timeout
Runtime log group missing while aws/spans exists
Malformed telemetry rows — this test also asserts logger warnings, which the fixture run() helper (silent logger, stdout only) can't capture
Same .send() seam, same real CoreClient, so it's the deterministic counterpart for the negative paths, not a weaker test. Added a comment saying so in ab60ff5.

Hweinstock · 2026-08-15T03:12:31Z

is this from this PR?


Run bun run format:check
$ prettier --check .
Checking formatting...
[warn] src/core/types.tsx
[warn] Code style issues found in the above file. Run Prettier with --write to fix.
error: script "format:check" exited with code 1

jariy17 · 2026-08-18T21:45:13Z

is this from this PR?

Yes — dropped semicolon in src/core/types.tsx:40. Restored it in ab60ff5; bun run format:check is clean now.

- Restore dropped semicolon in src/core/types.tsx that failed format:check - Explain why runWithRealCore stubs the SDK .send() seam inline instead of using the golden fixture harness (failure conditions the recorder can't provoke against a live account)

jariy17 · 2026-08-14T20:19:03Z

Follow-up to #1983: makes on-demand evaluation failures easier to classify and removes review-identified noise.

Changes

Missing runtime telemetry now throws ResourceNotFoundError.
Failed CloudWatch queries now throw CloudWatchQueryError with query status and ID metadata.
Malformed telemetry produces at most one warning per command.
Repeated validation tests now use test.each.
Redundant and drift-prone comments were removed.

Decisions

Fail fast: one session can require multiple Evaluate calls. Continuing after one fails could return incomplete results while appearing successful.
Keep results flat:context.spanContext.sessionId supports per-session filtering without changing the SDK response shape.
Defer a generic poller: extract one when another production caller needs the same behavior.
Keep window resolvers separate: batch feeds the API's DataSourceConfig, including CloudWatch log groups and onlineEvaluationConfigArn; on-demand feeds client-side Logs Insights.

Verification

Build, typecheck, lint, changed-file formatting, and focused tests pass.
Full suite: 1,373 tests pass.

codecov-commenter · 2026-08-14T20:21:27Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.13%. Comparing base (2e18071) to head (8ffdcb9).

Additional details and impacted files

@@ Coverage Diff @@## refactor #2007 +/- ##
============================================
+ Coverage 97.09% 97.13% +0.04% 
============================================
Files 381 381 Lines 22778 22786 +8 ============================================
+ Hits 22116 22134 +18 + Misses 662 652 -10

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Hweinstock

thanks for coming back to this! one question on the testing pattern, otherwise lgtm.

Hweinstock · 2026-08-15T03:05:51Z

 return { core, stdout: io.stdout(), stderr: io.stderr() };
 }

+async function runWithRealCore(options: LogsOptions, logger = createSilentLogger()) {


should this use the fixture pattern? Might be a little harder to trigger these cases, but seems possible unless i'm missing something.

Kept it on inline stubs on purpose. The fixture harness records against a live account; these cases can't be provoked there:
Query resolving Failed/Cancelled/Timeout
Runtime log group missing while aws/spans exists
Malformed telemetry rows — this test also asserts logger warnings, which the fixture run() helper (silent logger, stdout only) can't capture
Same .send() seam, same real CoreClient, so it's the deterministic counterpart for the negative paths, not a weaker test. Added a comment saying so in ab60ff5.

Hweinstock · 2026-08-15T03:12:31Z

is this from this PR?


Run bun run format:check
$ prettier --check .
Checking formatting...
[warn] src/core/types.tsx
[warn] Code style issues found in the above file. Run Prettier with --write to fix.
error: script "format:check" exited with code 1

jariy17 · 2026-08-18T21:45:13Z

is this from this PR?

Yes — dropped semicolon in src/core/types.tsx:40. Restored it in ab60ff5; bun run format:check is clean now.

- Restore dropped semicolon in src/core/types.tsx that failed format:check - Explain why runWithRealCore stubs the SDK .send() seam inline instead of using the golden fixture harness (failure conditions the recorder can't provoke against a live account)

jariy17 · 2026-08-14T20:19:03Z

Follow-up to #1983: makes on-demand evaluation failures easier to classify and removes review-identified noise.

Changes

Missing runtime telemetry now throws ResourceNotFoundError.
Failed CloudWatch queries now throw CloudWatchQueryError with query status and ID metadata.
Malformed telemetry produces at most one warning per command.
Repeated validation tests now use test.each.
Redundant and drift-prone comments were removed.

Decisions

Fail fast: one session can require multiple Evaluate calls. Continuing after one fails could return incomplete results while appearing successful.
Keep results flat:context.spanContext.sessionId supports per-session filtering without changing the SDK response shape.
Defer a generic poller: extract one when another production caller needs the same behavior.
Keep window resolvers separate: batch feeds the API's DataSourceConfig, including CloudWatch log groups and onlineEvaluationConfigArn; on-demand feeds client-side Logs Insights.

Verification

Build, typecheck, lint, changed-file formatting, and focused tests pass.
Full suite: 1,373 tests pass.

codecov-commenter · 2026-08-14T20:21:27Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.13%. Comparing base (2e18071) to head (8ffdcb9).

Additional details and impacted files

@@ Coverage Diff @@## refactor #2007 +/- ##
============================================
+ Coverage 97.09% 97.13% +0.04% 
============================================
Files 381 381 Lines 22778 22786 +8 ============================================
+ Hits 22116 22134 +18 + Misses 662 652 -10

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Hweinstock

thanks for coming back to this! one question on the testing pattern, otherwise lgtm.

Hweinstock · 2026-08-15T03:05:51Z

 return { core, stdout: io.stdout(), stderr: io.stderr() };
 }

+async function runWithRealCore(options: LogsOptions, logger = createSilentLogger()) {


should this use the fixture pattern? Might be a little harder to trigger these cases, but seems possible unless i'm missing something.

Kept it on inline stubs on purpose. The fixture harness records against a live account; these cases can't be provoked there:
Query resolving Failed/Cancelled/Timeout
Runtime log group missing while aws/spans exists
Malformed telemetry rows — this test also asserts logger warnings, which the fixture run() helper (silent logger, stdout only) can't capture
Same .send() seam, same real CoreClient, so it's the deterministic counterpart for the negative paths, not a weaker test. Added a comment saying so in ab60ff5.

Hweinstock · 2026-08-15T03:12:31Z

is this from this PR?


Run bun run format:check
$ prettier --check .
Checking formatting...
[warn] src/core/types.tsx
[warn] Code style issues found in the above file. Run Prettier with --write to fix.
error: script "format:check" exited with code 1

jariy17 · 2026-08-18T21:45:13Z

is this from this PR?

Yes — dropped semicolon in src/core/types.tsx:40. Restored it in ab60ff5; bun run format:check is clean now.

- Restore dropped semicolon in src/core/types.tsx that failed format:check - Explain why runWithRealCore stubs the SDK .send() seam inline instead of using the golden fixture harness (failure conditions the recorder can't provoke against a live account)

jariy17 · 2026-08-14T20:19:03Z

Follow-up to #1983: makes on-demand evaluation failures easier to classify and removes review-identified noise.

Changes

Missing runtime telemetry now throws ResourceNotFoundError.
Failed CloudWatch queries now throw CloudWatchQueryError with query status and ID metadata.
Malformed telemetry produces at most one warning per command.
Repeated validation tests now use test.each.
Redundant and drift-prone comments were removed.

Decisions

Fail fast: one session can require multiple Evaluate calls. Continuing after one fails could return incomplete results while appearing successful.
Keep results flat:context.spanContext.sessionId supports per-session filtering without changing the SDK response shape.
Defer a generic poller: extract one when another production caller needs the same behavior.
Keep window resolvers separate: batch feeds the API's DataSourceConfig, including CloudWatch log groups and onlineEvaluationConfigArn; on-demand feeds client-side Logs Insights.

Verification

Build, typecheck, lint, changed-file formatting, and focused tests pass.
Full suite: 1,373 tests pass.

codecov-commenter · 2026-08-14T20:21:27Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.13%. Comparing base (2e18071) to head (8ffdcb9).

Additional details and impacted files

@@ Coverage Diff @@## refactor #2007 +/- ##
============================================
+ Coverage 97.09% 97.13% +0.04% 
============================================
Files 381 381 Lines 22778 22786 +8 ============================================
+ Hits 22116 22134 +18 + Misses 662 652 -10

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Hweinstock

thanks for coming back to this! one question on the testing pattern, otherwise lgtm.

Hweinstock · 2026-08-15T03:05:51Z

 return { core, stdout: io.stdout(), stderr: io.stderr() };
 }

+async function runWithRealCore(options: LogsOptions, logger = createSilentLogger()) {


should this use the fixture pattern? Might be a little harder to trigger these cases, but seems possible unless i'm missing something.

Kept it on inline stubs on purpose. The fixture harness records against a live account; these cases can't be provoked there:
Query resolving Failed/Cancelled/Timeout
Runtime log group missing while aws/spans exists
Malformed telemetry rows — this test also asserts logger warnings, which the fixture run() helper (silent logger, stdout only) can't capture
Same .send() seam, same real CoreClient, so it's the deterministic counterpart for the negative paths, not a weaker test. Added a comment saying so in ab60ff5.

Hweinstock · 2026-08-15T03:12:31Z

is this from this PR?


Run bun run format:check
$ prettier --check .
Checking formatting...
[warn] src/core/types.tsx
[warn] Code style issues found in the above file. Run Prettier with --write to fix.
error: script "format:check" exited with code 1

jariy17 · 2026-08-18T21:45:13Z

is this from this PR?

Yes — dropped semicolon in src/core/types.tsx:40. Restored it in ab60ff5; bun run format:check is clean now.

- Restore dropped semicolon in src/core/types.tsx that failed format:check - Explain why runWithRealCore stubs the SDK .send() seam inline instead of using the golden fixture harness (failure conditions the recorder can't provoke against a live account)

jariy17 · 2026-08-14T20:19:03Z

Follow-up to #1983: makes on-demand evaluation failures easier to classify and removes review-identified noise.

Changes

Missing runtime telemetry now throws ResourceNotFoundError.
Failed CloudWatch queries now throw CloudWatchQueryError with query status and ID metadata.
Malformed telemetry produces at most one warning per command.
Repeated validation tests now use test.each.
Redundant and drift-prone comments were removed.

Decisions

Fail fast: one session can require multiple Evaluate calls. Continuing after one fails could return incomplete results while appearing successful.
Keep results flat:context.spanContext.sessionId supports per-session filtering without changing the SDK response shape.
Defer a generic poller: extract one when another production caller needs the same behavior.
Keep window resolvers separate: batch feeds the API's DataSourceConfig, including CloudWatch log groups and onlineEvaluationConfigArn; on-demand feeds client-side Logs Insights.

Verification

Build, typecheck, lint, changed-file formatting, and focused tests pass.
Full suite: 1,373 tests pass.

codecov-commenter · 2026-08-14T20:21:27Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.13%. Comparing base (2e18071) to head (8ffdcb9).

Additional details and impacted files

@@ Coverage Diff @@## refactor #2007 +/- ##
============================================
+ Coverage 97.09% 97.13% +0.04% 
============================================
Files 381 381 Lines 22778 22786 +8 ============================================
+ Hits 22116 22134 +18 + Misses 662 652 -10

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Hweinstock

thanks for coming back to this! one question on the testing pattern, otherwise lgtm.

Hweinstock · 2026-08-15T03:05:51Z

 return { core, stdout: io.stdout(), stderr: io.stderr() };
 }

+async function runWithRealCore(options: LogsOptions, logger = createSilentLogger()) {


should this use the fixture pattern? Might be a little harder to trigger these cases, but seems possible unless i'm missing something.

Kept it on inline stubs on purpose. The fixture harness records against a live account; these cases can't be provoked there:
Query resolving Failed/Cancelled/Timeout
Runtime log group missing while aws/spans exists
Malformed telemetry rows — this test also asserts logger warnings, which the fixture run() helper (silent logger, stdout only) can't capture
Same .send() seam, same real CoreClient, so it's the deterministic counterpart for the negative paths, not a weaker test. Added a comment saying so in ab60ff5.

Hweinstock · 2026-08-15T03:12:31Z

is this from this PR?


Run bun run format:check
$ prettier --check .
Checking formatting...
[warn] src/core/types.tsx
[warn] Code style issues found in the above file. Run Prettier with --write to fix.
error: script "format:check" exited with code 1

jariy17 · 2026-08-18T21:45:13Z

is this from this PR?

Yes — dropped semicolon in src/core/types.tsx:40. Restored it in ab60ff5; bun run format:check is clean now.

- Restore dropped semicolon in src/core/types.tsx that failed format:check - Explain why runWithRealCore stubs the SDK .send() seam inline instead of using the golden fixture harness (failure conditions the recorder can't provoke against a live account)

jariy17 · 2026-08-14T20:19:03Z

Follow-up to #1983: makes on-demand evaluation failures easier to classify and removes review-identified noise.

Changes

Missing runtime telemetry now throws ResourceNotFoundError.
Failed CloudWatch queries now throw CloudWatchQueryError with query status and ID metadata.
Malformed telemetry produces at most one warning per command.
Repeated validation tests now use test.each.
Redundant and drift-prone comments were removed.

Decisions

Fail fast: one session can require multiple Evaluate calls. Continuing after one fails could return incomplete results while appearing successful.
Keep results flat:context.spanContext.sessionId supports per-session filtering without changing the SDK response shape.
Defer a generic poller: extract one when another production caller needs the same behavior.
Keep window resolvers separate: batch feeds the API's DataSourceConfig, including CloudWatch log groups and onlineEvaluationConfigArn; on-demand feeds client-side Logs Insights.

Verification

Build, typecheck, lint, changed-file formatting, and focused tests pass.
Full suite: 1,373 tests pass.

codecov-commenter · 2026-08-14T20:21:27Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.13%. Comparing base (2e18071) to head (8ffdcb9).

Additional details and impacted files

@@ Coverage Diff @@## refactor #2007 +/- ##
============================================
+ Coverage 97.09% 97.13% +0.04% 
============================================
Files 381 381 Lines 22778 22786 +8 ============================================
+ Hits 22116 22134 +18 + Misses 662 652 -10

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Hweinstock

thanks for coming back to this! one question on the testing pattern, otherwise lgtm.

Hweinstock · 2026-08-15T03:05:51Z

 return { core, stdout: io.stdout(), stderr: io.stderr() };
 }

+async function runWithRealCore(options: LogsOptions, logger = createSilentLogger()) {


should this use the fixture pattern? Might be a little harder to trigger these cases, but seems possible unless i'm missing something.

Kept it on inline stubs on purpose. The fixture harness records against a live account; these cases can't be provoked there:
Query resolving Failed/Cancelled/Timeout
Runtime log group missing while aws/spans exists
Malformed telemetry rows — this test also asserts logger warnings, which the fixture run() helper (silent logger, stdout only) can't capture
Same .send() seam, same real CoreClient, so it's the deterministic counterpart for the negative paths, not a weaker test. Added a comment saying so in ab60ff5.

Hweinstock · 2026-08-15T03:12:31Z

is this from this PR?


Run bun run format:check
$ prettier --check .
Checking formatting...
[warn] src/core/types.tsx
[warn] Code style issues found in the above file. Run Prettier with --write to fix.
error: script "format:check" exited with code 1

jariy17 · 2026-08-18T21:45:13Z

is this from this PR?

Yes — dropped semicolon in src/core/types.tsx:40. Restored it in ab60ff5; bun run format:check is clean now.

- Restore dropped semicolon in src/core/types.tsx that failed format:check - Explain why runWithRealCore stubs the SDK .send() seam inline instead of using the golden fixture harness (failure conditions the recorder can't provoke against a live account)

jariy17 · 2026-08-14T20:19:03Z

Follow-up to #1983: makes on-demand evaluation failures easier to classify and removes review-identified noise.

Changes

Missing runtime telemetry now throws ResourceNotFoundError.
Failed CloudWatch queries now throw CloudWatchQueryError with query status and ID metadata.
Malformed telemetry produces at most one warning per command.
Repeated validation tests now use test.each.
Redundant and drift-prone comments were removed.

Decisions

Fail fast: one session can require multiple Evaluate calls. Continuing after one fails could return incomplete results while appearing successful.
Keep results flat:context.spanContext.sessionId supports per-session filtering without changing the SDK response shape.
Defer a generic poller: extract one when another production caller needs the same behavior.
Keep window resolvers separate: batch feeds the API's DataSourceConfig, including CloudWatch log groups and onlineEvaluationConfigArn; on-demand feeds client-side Logs Insights.

Verification

Build, typecheck, lint, changed-file formatting, and focused tests pass.
Full suite: 1,373 tests pass.

codecov-commenter · 2026-08-14T20:21:27Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.13%. Comparing base (2e18071) to head (8ffdcb9).

Additional details and impacted files

@@ Coverage Diff @@## refactor #2007 +/- ##
============================================
+ Coverage 97.09% 97.13% +0.04% 
============================================
Files 381 381 Lines 22778 22786 +8 ============================================
+ Hits 22116 22134 +18 + Misses 662 652 -10

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Hweinstock

thanks for coming back to this! one question on the testing pattern, otherwise lgtm.

Hweinstock · 2026-08-15T03:05:51Z

 return { core, stdout: io.stdout(), stderr: io.stderr() };
 }

+async function runWithRealCore(options: LogsOptions, logger = createSilentLogger()) {


should this use the fixture pattern? Might be a little harder to trigger these cases, but seems possible unless i'm missing something.

Kept it on inline stubs on purpose. The fixture harness records against a live account; these cases can't be provoked there:
Query resolving Failed/Cancelled/Timeout
Runtime log group missing while aws/spans exists
Malformed telemetry rows — this test also asserts logger warnings, which the fixture run() helper (silent logger, stdout only) can't capture
Same .send() seam, same real CoreClient, so it's the deterministic counterpart for the negative paths, not a weaker test. Added a comment saying so in ab60ff5.

Hweinstock · 2026-08-15T03:12:31Z

is this from this PR?


Run bun run format:check
$ prettier --check .
Checking formatting...
[warn] src/core/types.tsx
[warn] Code style issues found in the above file. Run Prettier with --write to fix.
error: script "format:check" exited with code 1

jariy17 · 2026-08-18T21:45:13Z

is this from this PR?

Yes — dropped semicolon in src/core/types.tsx:40. Restored it in ab60ff5; bun run format:check is clean now.

- Restore dropped semicolon in src/core/types.tsx that failed format:check - Explain why runWithRealCore stubs the SDK .send() seam inline instead of using the golden fixture harness (failure conditions the recorder can't provoke against a live account)

jariy17 · 2026-08-14T20:19:03Z

Follow-up to #1983: makes on-demand evaluation failures easier to classify and removes review-identified noise.

Changes

Missing runtime telemetry now throws ResourceNotFoundError.
Failed CloudWatch queries now throw CloudWatchQueryError with query status and ID metadata.
Malformed telemetry produces at most one warning per command.
Repeated validation tests now use test.each.
Redundant and drift-prone comments were removed.

Decisions

Fail fast: one session can require multiple Evaluate calls. Continuing after one fails could return incomplete results while appearing successful.
Keep results flat:context.spanContext.sessionId supports per-session filtering without changing the SDK response shape.
Defer a generic poller: extract one when another production caller needs the same behavior.
Keep window resolvers separate: batch feeds the API's DataSourceConfig, including CloudWatch log groups and onlineEvaluationConfigArn; on-demand feeds client-side Logs Insights.

Verification

Build, typecheck, lint, changed-file formatting, and focused tests pass.
Full suite: 1,373 tests pass.

codecov-commenter · 2026-08-14T20:21:27Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.13%. Comparing base (2e18071) to head (8ffdcb9).

Additional details and impacted files

@@ Coverage Diff @@## refactor #2007 +/- ##
============================================
+ Coverage 97.09% 97.13% +0.04% 
============================================
Files 381 381 Lines 22778 22786 +8 ============================================
+ Hits 22116 22134 +18 + Misses 662 652 -10

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Hweinstock

thanks for coming back to this! one question on the testing pattern, otherwise lgtm.

Hweinstock · 2026-08-15T03:05:51Z

 return { core, stdout: io.stdout(), stderr: io.stderr() };
 }

+async function runWithRealCore(options: LogsOptions, logger = createSilentLogger()) {


should this use the fixture pattern? Might be a little harder to trigger these cases, but seems possible unless i'm missing something.

Kept it on inline stubs on purpose. The fixture harness records against a live account; these cases can't be provoked there:
Query resolving Failed/Cancelled/Timeout
Runtime log group missing while aws/spans exists
Malformed telemetry rows — this test also asserts logger warnings, which the fixture run() helper (silent logger, stdout only) can't capture
Same .send() seam, same real CoreClient, so it's the deterministic counterpart for the negative paths, not a weaker test. Added a comment saying so in ab60ff5.

Hweinstock · 2026-08-15T03:12:31Z

is this from this PR?


Run bun run format:check
$ prettier --check .
Checking formatting...
[warn] src/core/types.tsx
[warn] Code style issues found in the above file. Run Prettier with --write to fix.
error: script "format:check" exited with code 1

jariy17 · 2026-08-18T21:45:13Z

is this from this PR?

Yes — dropped semicolon in src/core/types.tsx:40. Restored it in ab60ff5; bun run format:check is clean now.

- Restore dropped semicolon in src/core/types.tsx that failed format:check - Explain why runWithRealCore stubs the SDK .send() seam inline instead of using the golden fixture harness (failure conditions the recorder can't provoke against a live account)

github-actionsBot added the agentcore-harness-reviewing AgentCore Harness review in progress label Aug 14, 2026

github-actionsBot removed the agentcore-harness-reviewing AgentCore Harness review in progress label Aug 14, 2026

Hweinstock reviewed Aug 15, 2026
View reviewed changes

jariy17 added 3 commits August 18, 2026 22:04

refactor(eval): address on-demand review feedback
3762c65

test(eval): consolidate on-demand regression coverage
1676291

jariy17 force-pushed the followup/eval-ondemand-review-feedback branch from ab60ff5 to 8ffdcb9Compare August 18, 2026 22:06

Hweinstock approved these changes Aug 18, 2026
View reviewed changes

aidandaly24 approved these changes Aug 18, 2026
View reviewed changes

jariy17 merged commit ee854eb into aws:refactorAug 18, 2026
8 of 11 checks passed