fix: add batch eval and recommendation IAM permissions to docs by notgitika · Pull Request #1113 · aws/agentcore-cli

notgitika · 2026-05-04T21:33:27Z

Summary

Adds missing IAM permissions to docs policy files for batch evaluation and recommendations:

Add BatchEvalAndRecommendations statement to iam-policy-user.json (9 actions)
Add CloudWatch Logs write permissions to LogsStreamingAndSearch (DescribeLogGroups, CreateLogGroup, CreateLogStream, PutLogEvents, PutRetentionPolicy)
Update PERMISSIONS.md with batch eval/recommendations reference section and logging entries

Also applied logs:PutRetentionPolicy to the e2e-github-actions role in the test account.

Context

E2E batch eval tests were failing with 403 on PutRetentionPolicy -- the batch eval API sets a retention policy on its results log group.

Test plan

iam-policy-user.json is valid JSON
prettier passes
Applied PutRetentionPolicy to e2e-github-actions role
Re-run E2E to confirm batch eval tests pass

…sions to docs

agentcore-cli-automation · 2026-05-04T21:36:53Z

Thanks for adding these permissions. One issue worth addressing before merge:

The CLI Commands column in PERMISSIONS.md references commands that don't exist in the CLI today.

In the new "Batch evaluation and recommendations" table:

run batch-eval — no such command. run only has the eval subcommand (see src/cli/commands/run/command.tsx).
run recommend — no such command.
evals history is listed for bedrock-agentcore:ListBatchEvaluations, but evals history is a local-only command that reads from disk via handleListEvalRuns → listEvalRuns() (see src/cli/operations/eval/list-eval-runs.ts and src/cli/commands/eval/command.tsx). It makes no AWS API call, so it doesn't require ListBatchEvaluations.

I also couldn't find any references to StartBatchEvaluation, StartRecommendation, etc. anywhere in src/ or the e2e tests — the run eval code path uses the Evaluate API (already covered).

If the service is invoking batch eval / recommendation APIs server-side on behalf of the user when run eval is called (which would explain the PutRetentionPolicy 403 you mentioned in the description), it might be worth:

Options to address this:

Update the CLI Commands column entries to point to the actual command that triggers these permissions (e.g. run evals if that's the command that indirectly causes batch eval invocations server-side), and update the Purpose column to explain it's service-invoked on behalf of the caller.
Drop the Batch evaluation and recommendations section from PERMISSIONS.md and the BatchEvalAndRecommendations statement from iam-policy-user.json for now if no CLI command actually calls these APIs — and only keep the logs permissions that are actually failing (PutRetentionPolicy, etc.).
If these commands are planned/upcoming, either add them in this PR or gate the doc changes until the commands land, so the docs don't advertise commands users can't run.

The logs:* additions (PutRetentionPolicy, CreateLogGroup, CreateLogStream, PutLogEvents, DescribeLogGroups) look fine as long as the batch-eval-via-service-principal explanation holds, though the CLI Commands column still lists the non-existent run batch-eval for most of them and should be fixed the same way.

github-actions · 2026-05-04T21:37:13Z

Coverage Report

Status	Category	Percentage	Covered / Total
🔵	Lines	43.04%	8990 / 20883
🔵	Statements	42.33%	9546 / 22550
🔵	Functions	39.91%	1552 / 3888
🔵	Branches	39.92%	5794 / 14511

Generated in workflow #2378 for commit 1b9ef28 by the Vitest Coverage Report Action

Cherry-picks non-breaking changes from main: - docs: add telemetry instrumentation guide (#1197) - chore: replace PAT tokens with GitHub App token (#1198) - fix: add batch eval, recommendation, CloudWatch Logs permissions (#1113) - feat(evaluator): add kmsKeyArn support (#994) - chore: replace github.token with GitHub App token (#1210) - fix: sync-preview workflow rewrite (#1078) Skipped (requires dedicated effort due to Result type refactor): - refactor: unify result types with Result<T, E> (#1125) - feat: instrument telemetry for create/deploy (#1202, #1206) - feat: record command attrs on failure (#1204)

* docs: add telemetry instrumentation guide (#1197) * chore: replace PAT tokens with GitHub App token (#1198) Replace secrets.PAT_TOKEN and secrets.AUTOMATION_ACCOUNT_PAT_TOKEN with short-lived tokens generated by the agentcore-devx-automation GitHub App (ID: 3637953) via actions/create-github-app-token@v1. This improves security by using ephemeral tokens scoped to the installation rather than long-lived personal access tokens. Requires adding repo variable APP_ID=3637953 and repo secret APP_PRIVATE_KEY with the app's RSA private key. * fix: add batch eval, recommendation, and CloudWatch Logs write permissions to docs (#1113) * feat: instrument telemetry for create command (CLI + TUI) (#1202) Add telemetry recording to the create command in both CLI and TUI paths: - CLI: wrap handleCreateCLI with runCliCommand to emit CreateAttrs on success/failure - TUI: wrap useCreateFlow's run() with withCommandRunTelemetry - Add telemetry assertions to existing integration tests (frameworks + edge cases) * chore: replace all github.token/GITHUB_TOKEN with GitHub App token Replaces all occurrences of \${{ github.token }} and \${{ secrets.GITHUB_TOKEN }} across .github/workflows/ with a per-job GitHub App token generated via actions/create-github-app-token@v1 using vars.APP_ID and secrets.APP_PRIVATE_KEY. * fix: bump versions to resolve security audit failure * revert: keep pr-title.yml using GITHUB_TOKEN (read-only access sufficient) * feat(evaluator): add kmsKeyArn support for custom evaluator (#994) * feat(evaluator): Add kmsKeyArn support for custom evaluator * fix: sync package-lock.json with package.json The lock file was out of sync after dependency bumps on main were merged, causing npm ci to fail in CI. * fix: revert unrelated dep bumps and fix formatting Reverts @opentelemetry/exporter-metrics-otlp-http ^0.217.0 back to ^0.214.0 and secretlint ^13.0.0 back to ^12.2.0 — these were accidentally included in the feature commit from unmerged dependabot PRs and introduce high-severity protobufjs vulnerabilities. Restores fast-xml-parser and @aws-sdk/xml-builder overrides that were also inadvertently removed. Fixes Prettier formatting on agentcore-project.ts import lines. * fix: sync package-lock.json with updated dependencies --------- Co-authored-by: notgitika <gitijh@gmail.com> * refactor: unify result types with discriminated Result<T, E> union (#1125) * refactor: unify result types with discriminated Result<T, E> union Introduce a shared Result<T, E> type (inspired by Rust's Result) that replaces ad-hoc { success: boolean; error?: string } patterns across the codebase. Key changes: - Add src/lib/types.ts with Result<T, E> discriminated union type - Add toError() helper in src/cli/errors.ts for catch blocks - Migrate all command, operation, and primitive result types to Result<T> - Error field is now Error (not string) on the failure branch - Data fields only exist on the success branch (proper narrowing) - Update all consumers to narrow before accessing branch-specific fields - Update test assertions to match new Error objects and add narrowing * docs: update AGENTS.md and telemetry README to reflect Result<T, E> type * feat: record command attrs on telemetry failure via fallbackAttrs (#1204) * feat: record command attrs on telemetry failure via fallbackAttrs Add optional fallbackAttrs parameter to client.withCommandRun so command-specific attributes are recorded even when the callback throws. - client.ts: accept fallbackAttrs, use on failure instead of {} - client.ts: run resilientParse on all non-empty attrs (not just success) - cli-command-run.ts: withCommandRunTelemetry passes attrs as fallbackAttrs - cli-command-run.ts: runCliCommand accepts optional knownAttrs param - command.tsx: extract knownAttrs upfront, pass to runCliCommand - client.test.ts: add unit tests for fallbackAttrs behavior - create-edge-cases.test.ts: assert attrs present on failure entry * chore: rebase onto mainline * fix: sync-preview pushes directly on clean merge, PRs only on conflict (#1078) * fix: sync-preview workflow restores version instead of ignoring files Instead of keeping preview's entire package.json/package-lock.json (which discards new deps, scripts, etc. from main), accept main's content and surgically restore only the version field to preview's value after merge. * fix: push directly to preview on clean merge via GitHub App bypass Use agentcore-devx-automation app token to bypass branch protection and push directly when the merge is clean (or only version conflicts). Only creates a PR when there are real conflicts in other files. * chore: use app-slug instead of app-id for token generation * fix: address review feedback on sync-preview workflow - Pass PREVIEW_VERSION via env var instead of string interpolation in node -e scripts (safer against special chars) - Make git add of package-lock.json conditional on file existence to match the earlier -f guard - Replace loose title search for dedup with headRefName prefix filter to avoid false positives from unrelated PRs - Clarify why package.json/package-lock.json are special-cased (preview carries a different version string that needs preserving) * fix: restore preview-owned files after sync merge Adds a step to restore schemas/agentcore.schema.v1.json and CHANGELOG.md to preview's versions after merging main. These files are auto-generated during preview releases — schema-check CI rejects direct modifications to schemas/, and CHANGELOG.md tracks preview releases separately. * fix: use app-id instead of app-slug for GitHub App token Aligns with the pattern in PR #1210 and ci-failure-issue.yml. * feat: instrument telemetry for deploy command (CLI + TUI) (#1206) * chore: sync safe commits from main into preview Cherry-picks non-breaking changes from main: - docs: add telemetry instrumentation guide (#1197) - chore: replace PAT tokens with GitHub App token (#1198) - fix: add batch eval, recommendation, CloudWatch Logs permissions (#1113) - feat(evaluator): add kmsKeyArn support (#994) - chore: replace github.token with GitHub App token (#1210) - fix: sync-preview workflow rewrite (#1078) Skipped (requires dedicated effort due to Result type refactor): - refactor: unify result types with Result<T, E> (#1125) - feat: instrument telemetry for create/deploy (#1202, #1206) - feat: record command attrs on failure (#1204) * chore: merge main into preview with Result refactor Brings in all remaining main commits including: - refactor: unify result types with Result<T, E> (#1125) - feat: instrument telemetry for deploy command (#1206) - feat: record command attrs on failure (#1204) - feat: instrument telemetry for create command (#1202) Adapts preview's deploy flow to use OperationResult with string errors for compatibility with the telemetry layer. --------- Co-authored-by: Hweinstock <42325418+Hweinstock@users.noreply.github.com> Co-authored-by: Aidan Daly <99039782+aidandaly24@users.noreply.github.com> Co-authored-by: Gitika <53349492+notgitika@users.noreply.github.com> Co-authored-by: Aidan Daly <aidandal@amazon.com> Co-authored-by: Harrison Weinstock <hkobew@amazon.com> Co-authored-by: aws-aditya21 <saivenki@amazon.com> Co-authored-by: notgitika <gitijh@gmail.com>

notgitika · 2026-05-04T21:33:27Z

Summary

Adds missing IAM permissions to docs policy files for batch evaluation and recommendations:

Add BatchEvalAndRecommendations statement to iam-policy-user.json (9 actions)
Add CloudWatch Logs write permissions to LogsStreamingAndSearch (DescribeLogGroups, CreateLogGroup, CreateLogStream, PutLogEvents, PutRetentionPolicy)
Update PERMISSIONS.md with batch eval/recommendations reference section and logging entries

Also applied logs:PutRetentionPolicy to the e2e-github-actions role in the test account.

Context

E2E batch eval tests were failing with 403 on PutRetentionPolicy -- the batch eval API sets a retention policy on its results log group.

Test plan

iam-policy-user.json is valid JSON
prettier passes
Applied PutRetentionPolicy to e2e-github-actions role
Re-run E2E to confirm batch eval tests pass

…sions to docs

agentcore-cli-automation · 2026-05-04T21:36:53Z

Thanks for adding these permissions. One issue worth addressing before merge:

The CLI Commands column in PERMISSIONS.md references commands that don't exist in the CLI today.

In the new "Batch evaluation and recommendations" table:

run batch-eval — no such command. run only has the eval subcommand (see src/cli/commands/run/command.tsx).
run recommend — no such command.
evals history is listed for bedrock-agentcore:ListBatchEvaluations, but evals history is a local-only command that reads from disk via handleListEvalRuns → listEvalRuns() (see src/cli/operations/eval/list-eval-runs.ts and src/cli/commands/eval/command.tsx). It makes no AWS API call, so it doesn't require ListBatchEvaluations.

I also couldn't find any references to StartBatchEvaluation, StartRecommendation, etc. anywhere in src/ or the e2e tests — the run eval code path uses the Evaluate API (already covered).

If the service is invoking batch eval / recommendation APIs server-side on behalf of the user when run eval is called (which would explain the PutRetentionPolicy 403 you mentioned in the description), it might be worth:

Options to address this:

Update the CLI Commands column entries to point to the actual command that triggers these permissions (e.g. run evals if that's the command that indirectly causes batch eval invocations server-side), and update the Purpose column to explain it's service-invoked on behalf of the caller.
Drop the Batch evaluation and recommendations section from PERMISSIONS.md and the BatchEvalAndRecommendations statement from iam-policy-user.json for now if no CLI command actually calls these APIs — and only keep the logs permissions that are actually failing (PutRetentionPolicy, etc.).
If these commands are planned/upcoming, either add them in this PR or gate the doc changes until the commands land, so the docs don't advertise commands users can't run.

The logs:* additions (PutRetentionPolicy, CreateLogGroup, CreateLogStream, PutLogEvents, DescribeLogGroups) look fine as long as the batch-eval-via-service-principal explanation holds, though the CLI Commands column still lists the non-existent run batch-eval for most of them and should be fixed the same way.

github-actions · 2026-05-04T21:37:13Z

Coverage Report

Status	Category	Percentage	Covered / Total
🔵	Lines	43.04%	8990 / 20883
🔵	Statements	42.33%	9546 / 22550
🔵	Functions	39.91%	1552 / 3888
🔵	Branches	39.92%	5794 / 14511

Generated in workflow #2378 for commit 1b9ef28 by the Vitest Coverage Report Action

Cherry-picks non-breaking changes from main: - docs: add telemetry instrumentation guide (#1197) - chore: replace PAT tokens with GitHub App token (#1198) - fix: add batch eval, recommendation, CloudWatch Logs permissions (#1113) - feat(evaluator): add kmsKeyArn support (#994) - chore: replace github.token with GitHub App token (#1210) - fix: sync-preview workflow rewrite (#1078) Skipped (requires dedicated effort due to Result type refactor): - refactor: unify result types with Result<T, E> (#1125) - feat: instrument telemetry for create/deploy (#1202, #1206) - feat: record command attrs on failure (#1204)

* docs: add telemetry instrumentation guide (#1197) * chore: replace PAT tokens with GitHub App token (#1198) Replace secrets.PAT_TOKEN and secrets.AUTOMATION_ACCOUNT_PAT_TOKEN with short-lived tokens generated by the agentcore-devx-automation GitHub App (ID: 3637953) via actions/create-github-app-token@v1. This improves security by using ephemeral tokens scoped to the installation rather than long-lived personal access tokens. Requires adding repo variable APP_ID=3637953 and repo secret APP_PRIVATE_KEY with the app's RSA private key. * fix: add batch eval, recommendation, and CloudWatch Logs write permissions to docs (#1113) * feat: instrument telemetry for create command (CLI + TUI) (#1202) Add telemetry recording to the create command in both CLI and TUI paths: - CLI: wrap handleCreateCLI with runCliCommand to emit CreateAttrs on success/failure - TUI: wrap useCreateFlow's run() with withCommandRunTelemetry - Add telemetry assertions to existing integration tests (frameworks + edge cases) * chore: replace all github.token/GITHUB_TOKEN with GitHub App token Replaces all occurrences of \${{ github.token }} and \${{ secrets.GITHUB_TOKEN }} across .github/workflows/ with a per-job GitHub App token generated via actions/create-github-app-token@v1 using vars.APP_ID and secrets.APP_PRIVATE_KEY. * fix: bump versions to resolve security audit failure * revert: keep pr-title.yml using GITHUB_TOKEN (read-only access sufficient) * feat(evaluator): add kmsKeyArn support for custom evaluator (#994) * feat(evaluator): Add kmsKeyArn support for custom evaluator * fix: sync package-lock.json with package.json The lock file was out of sync after dependency bumps on main were merged, causing npm ci to fail in CI. * fix: revert unrelated dep bumps and fix formatting Reverts @opentelemetry/exporter-metrics-otlp-http ^0.217.0 back to ^0.214.0 and secretlint ^13.0.0 back to ^12.2.0 — these were accidentally included in the feature commit from unmerged dependabot PRs and introduce high-severity protobufjs vulnerabilities. Restores fast-xml-parser and @aws-sdk/xml-builder overrides that were also inadvertently removed. Fixes Prettier formatting on agentcore-project.ts import lines. * fix: sync package-lock.json with updated dependencies --------- Co-authored-by: notgitika <gitijh@gmail.com> * refactor: unify result types with discriminated Result<T, E> union (#1125) * refactor: unify result types with discriminated Result<T, E> union Introduce a shared Result<T, E> type (inspired by Rust's Result) that replaces ad-hoc { success: boolean; error?: string } patterns across the codebase. Key changes: - Add src/lib/types.ts with Result<T, E> discriminated union type - Add toError() helper in src/cli/errors.ts for catch blocks - Migrate all command, operation, and primitive result types to Result<T> - Error field is now Error (not string) on the failure branch - Data fields only exist on the success branch (proper narrowing) - Update all consumers to narrow before accessing branch-specific fields - Update test assertions to match new Error objects and add narrowing * docs: update AGENTS.md and telemetry README to reflect Result<T, E> type * feat: record command attrs on telemetry failure via fallbackAttrs (#1204) * feat: record command attrs on telemetry failure via fallbackAttrs Add optional fallbackAttrs parameter to client.withCommandRun so command-specific attributes are recorded even when the callback throws. - client.ts: accept fallbackAttrs, use on failure instead of {} - client.ts: run resilientParse on all non-empty attrs (not just success) - cli-command-run.ts: withCommandRunTelemetry passes attrs as fallbackAttrs - cli-command-run.ts: runCliCommand accepts optional knownAttrs param - command.tsx: extract knownAttrs upfront, pass to runCliCommand - client.test.ts: add unit tests for fallbackAttrs behavior - create-edge-cases.test.ts: assert attrs present on failure entry * chore: rebase onto mainline * fix: sync-preview pushes directly on clean merge, PRs only on conflict (#1078) * fix: sync-preview workflow restores version instead of ignoring files Instead of keeping preview's entire package.json/package-lock.json (which discards new deps, scripts, etc. from main), accept main's content and surgically restore only the version field to preview's value after merge. * fix: push directly to preview on clean merge via GitHub App bypass Use agentcore-devx-automation app token to bypass branch protection and push directly when the merge is clean (or only version conflicts). Only creates a PR when there are real conflicts in other files. * chore: use app-slug instead of app-id for token generation * fix: address review feedback on sync-preview workflow - Pass PREVIEW_VERSION via env var instead of string interpolation in node -e scripts (safer against special chars) - Make git add of package-lock.json conditional on file existence to match the earlier -f guard - Replace loose title search for dedup with headRefName prefix filter to avoid false positives from unrelated PRs - Clarify why package.json/package-lock.json are special-cased (preview carries a different version string that needs preserving) * fix: restore preview-owned files after sync merge Adds a step to restore schemas/agentcore.schema.v1.json and CHANGELOG.md to preview's versions after merging main. These files are auto-generated during preview releases — schema-check CI rejects direct modifications to schemas/, and CHANGELOG.md tracks preview releases separately. * fix: use app-id instead of app-slug for GitHub App token Aligns with the pattern in PR #1210 and ci-failure-issue.yml. * feat: instrument telemetry for deploy command (CLI + TUI) (#1206) * chore: sync safe commits from main into preview Cherry-picks non-breaking changes from main: - docs: add telemetry instrumentation guide (#1197) - chore: replace PAT tokens with GitHub App token (#1198) - fix: add batch eval, recommendation, CloudWatch Logs permissions (#1113) - feat(evaluator): add kmsKeyArn support (#994) - chore: replace github.token with GitHub App token (#1210) - fix: sync-preview workflow rewrite (#1078) Skipped (requires dedicated effort due to Result type refactor): - refactor: unify result types with Result<T, E> (#1125) - feat: instrument telemetry for create/deploy (#1202, #1206) - feat: record command attrs on failure (#1204) * chore: merge main into preview with Result refactor Brings in all remaining main commits including: - refactor: unify result types with Result<T, E> (#1125) - feat: instrument telemetry for deploy command (#1206) - feat: record command attrs on failure (#1204) - feat: instrument telemetry for create command (#1202) Adapts preview's deploy flow to use OperationResult with string errors for compatibility with the telemetry layer. --------- Co-authored-by: Hweinstock <42325418+Hweinstock@users.noreply.github.com> Co-authored-by: Aidan Daly <99039782+aidandaly24@users.noreply.github.com> Co-authored-by: Gitika <53349492+notgitika@users.noreply.github.com> Co-authored-by: Aidan Daly <aidandal@amazon.com> Co-authored-by: Harrison Weinstock <hkobew@amazon.com> Co-authored-by: aws-aditya21 <saivenki@amazon.com> Co-authored-by: notgitika <gitijh@gmail.com>

notgitika · 2026-05-04T21:33:27Z

Summary

Adds missing IAM permissions to docs policy files for batch evaluation and recommendations:

Add BatchEvalAndRecommendations statement to iam-policy-user.json (9 actions)
Add CloudWatch Logs write permissions to LogsStreamingAndSearch (DescribeLogGroups, CreateLogGroup, CreateLogStream, PutLogEvents, PutRetentionPolicy)
Update PERMISSIONS.md with batch eval/recommendations reference section and logging entries

Also applied logs:PutRetentionPolicy to the e2e-github-actions role in the test account.

Context

E2E batch eval tests were failing with 403 on PutRetentionPolicy -- the batch eval API sets a retention policy on its results log group.

Test plan

iam-policy-user.json is valid JSON
prettier passes
Applied PutRetentionPolicy to e2e-github-actions role
Re-run E2E to confirm batch eval tests pass

…sions to docs

agentcore-cli-automation · 2026-05-04T21:36:53Z

Thanks for adding these permissions. One issue worth addressing before merge:

The CLI Commands column in PERMISSIONS.md references commands that don't exist in the CLI today.

In the new "Batch evaluation and recommendations" table:

run batch-eval — no such command. run only has the eval subcommand (see src/cli/commands/run/command.tsx).
run recommend — no such command.
evals history is listed for bedrock-agentcore:ListBatchEvaluations, but evals history is a local-only command that reads from disk via handleListEvalRuns → listEvalRuns() (see src/cli/operations/eval/list-eval-runs.ts and src/cli/commands/eval/command.tsx). It makes no AWS API call, so it doesn't require ListBatchEvaluations.

I also couldn't find any references to StartBatchEvaluation, StartRecommendation, etc. anywhere in src/ or the e2e tests — the run eval code path uses the Evaluate API (already covered).

If the service is invoking batch eval / recommendation APIs server-side on behalf of the user when run eval is called (which would explain the PutRetentionPolicy 403 you mentioned in the description), it might be worth:

Options to address this:

Update the CLI Commands column entries to point to the actual command that triggers these permissions (e.g. run evals if that's the command that indirectly causes batch eval invocations server-side), and update the Purpose column to explain it's service-invoked on behalf of the caller.
Drop the Batch evaluation and recommendations section from PERMISSIONS.md and the BatchEvalAndRecommendations statement from iam-policy-user.json for now if no CLI command actually calls these APIs — and only keep the logs permissions that are actually failing (PutRetentionPolicy, etc.).
If these commands are planned/upcoming, either add them in this PR or gate the doc changes until the commands land, so the docs don't advertise commands users can't run.

The logs:* additions (PutRetentionPolicy, CreateLogGroup, CreateLogStream, PutLogEvents, DescribeLogGroups) look fine as long as the batch-eval-via-service-principal explanation holds, though the CLI Commands column still lists the non-existent run batch-eval for most of them and should be fixed the same way.

github-actions · 2026-05-04T21:37:13Z

Coverage Report

Status	Category	Percentage	Covered / Total
🔵	Lines	43.04%	8990 / 20883
🔵	Statements	42.33%	9546 / 22550
🔵	Functions	39.91%	1552 / 3888
🔵	Branches	39.92%	5794 / 14511

Generated in workflow #2378 for commit 1b9ef28 by the Vitest Coverage Report Action

Cherry-picks non-breaking changes from main: - docs: add telemetry instrumentation guide (#1197) - chore: replace PAT tokens with GitHub App token (#1198) - fix: add batch eval, recommendation, CloudWatch Logs permissions (#1113) - feat(evaluator): add kmsKeyArn support (#994) - chore: replace github.token with GitHub App token (#1210) - fix: sync-preview workflow rewrite (#1078) Skipped (requires dedicated effort due to Result type refactor): - refactor: unify result types with Result<T, E> (#1125) - feat: instrument telemetry for create/deploy (#1202, #1206) - feat: record command attrs on failure (#1204)

* docs: add telemetry instrumentation guide (#1197) * chore: replace PAT tokens with GitHub App token (#1198) Replace secrets.PAT_TOKEN and secrets.AUTOMATION_ACCOUNT_PAT_TOKEN with short-lived tokens generated by the agentcore-devx-automation GitHub App (ID: 3637953) via actions/create-github-app-token@v1. This improves security by using ephemeral tokens scoped to the installation rather than long-lived personal access tokens. Requires adding repo variable APP_ID=3637953 and repo secret APP_PRIVATE_KEY with the app's RSA private key. * fix: add batch eval, recommendation, and CloudWatch Logs write permissions to docs (#1113) * feat: instrument telemetry for create command (CLI + TUI) (#1202) Add telemetry recording to the create command in both CLI and TUI paths: - CLI: wrap handleCreateCLI with runCliCommand to emit CreateAttrs on success/failure - TUI: wrap useCreateFlow's run() with withCommandRunTelemetry - Add telemetry assertions to existing integration tests (frameworks + edge cases) * chore: replace all github.token/GITHUB_TOKEN with GitHub App token Replaces all occurrences of \${{ github.token }} and \${{ secrets.GITHUB_TOKEN }} across .github/workflows/ with a per-job GitHub App token generated via actions/create-github-app-token@v1 using vars.APP_ID and secrets.APP_PRIVATE_KEY. * fix: bump versions to resolve security audit failure * revert: keep pr-title.yml using GITHUB_TOKEN (read-only access sufficient) * feat(evaluator): add kmsKeyArn support for custom evaluator (#994) * feat(evaluator): Add kmsKeyArn support for custom evaluator * fix: sync package-lock.json with package.json The lock file was out of sync after dependency bumps on main were merged, causing npm ci to fail in CI. * fix: revert unrelated dep bumps and fix formatting Reverts @opentelemetry/exporter-metrics-otlp-http ^0.217.0 back to ^0.214.0 and secretlint ^13.0.0 back to ^12.2.0 — these were accidentally included in the feature commit from unmerged dependabot PRs and introduce high-severity protobufjs vulnerabilities. Restores fast-xml-parser and @aws-sdk/xml-builder overrides that were also inadvertently removed. Fixes Prettier formatting on agentcore-project.ts import lines. * fix: sync package-lock.json with updated dependencies --------- Co-authored-by: notgitika <gitijh@gmail.com> * refactor: unify result types with discriminated Result<T, E> union (#1125) * refactor: unify result types with discriminated Result<T, E> union Introduce a shared Result<T, E> type (inspired by Rust's Result) that replaces ad-hoc { success: boolean; error?: string } patterns across the codebase. Key changes: - Add src/lib/types.ts with Result<T, E> discriminated union type - Add toError() helper in src/cli/errors.ts for catch blocks - Migrate all command, operation, and primitive result types to Result<T> - Error field is now Error (not string) on the failure branch - Data fields only exist on the success branch (proper narrowing) - Update all consumers to narrow before accessing branch-specific fields - Update test assertions to match new Error objects and add narrowing * docs: update AGENTS.md and telemetry README to reflect Result<T, E> type * feat: record command attrs on telemetry failure via fallbackAttrs (#1204) * feat: record command attrs on telemetry failure via fallbackAttrs Add optional fallbackAttrs parameter to client.withCommandRun so command-specific attributes are recorded even when the callback throws. - client.ts: accept fallbackAttrs, use on failure instead of {} - client.ts: run resilientParse on all non-empty attrs (not just success) - cli-command-run.ts: withCommandRunTelemetry passes attrs as fallbackAttrs - cli-command-run.ts: runCliCommand accepts optional knownAttrs param - command.tsx: extract knownAttrs upfront, pass to runCliCommand - client.test.ts: add unit tests for fallbackAttrs behavior - create-edge-cases.test.ts: assert attrs present on failure entry * chore: rebase onto mainline * fix: sync-preview pushes directly on clean merge, PRs only on conflict (#1078) * fix: sync-preview workflow restores version instead of ignoring files Instead of keeping preview's entire package.json/package-lock.json (which discards new deps, scripts, etc. from main), accept main's content and surgically restore only the version field to preview's value after merge. * fix: push directly to preview on clean merge via GitHub App bypass Use agentcore-devx-automation app token to bypass branch protection and push directly when the merge is clean (or only version conflicts). Only creates a PR when there are real conflicts in other files. * chore: use app-slug instead of app-id for token generation * fix: address review feedback on sync-preview workflow - Pass PREVIEW_VERSION via env var instead of string interpolation in node -e scripts (safer against special chars) - Make git add of package-lock.json conditional on file existence to match the earlier -f guard - Replace loose title search for dedup with headRefName prefix filter to avoid false positives from unrelated PRs - Clarify why package.json/package-lock.json are special-cased (preview carries a different version string that needs preserving) * fix: restore preview-owned files after sync merge Adds a step to restore schemas/agentcore.schema.v1.json and CHANGELOG.md to preview's versions after merging main. These files are auto-generated during preview releases — schema-check CI rejects direct modifications to schemas/, and CHANGELOG.md tracks preview releases separately. * fix: use app-id instead of app-slug for GitHub App token Aligns with the pattern in PR #1210 and ci-failure-issue.yml. * feat: instrument telemetry for deploy command (CLI + TUI) (#1206) * chore: sync safe commits from main into preview Cherry-picks non-breaking changes from main: - docs: add telemetry instrumentation guide (#1197) - chore: replace PAT tokens with GitHub App token (#1198) - fix: add batch eval, recommendation, CloudWatch Logs permissions (#1113) - feat(evaluator): add kmsKeyArn support (#994) - chore: replace github.token with GitHub App token (#1210) - fix: sync-preview workflow rewrite (#1078) Skipped (requires dedicated effort due to Result type refactor): - refactor: unify result types with Result<T, E> (#1125) - feat: instrument telemetry for create/deploy (#1202, #1206) - feat: record command attrs on failure (#1204) * chore: merge main into preview with Result refactor Brings in all remaining main commits including: - refactor: unify result types with Result<T, E> (#1125) - feat: instrument telemetry for deploy command (#1206) - feat: record command attrs on failure (#1204) - feat: instrument telemetry for create command (#1202) Adapts preview's deploy flow to use OperationResult with string errors for compatibility with the telemetry layer. --------- Co-authored-by: Hweinstock <42325418+Hweinstock@users.noreply.github.com> Co-authored-by: Aidan Daly <99039782+aidandaly24@users.noreply.github.com> Co-authored-by: Gitika <53349492+notgitika@users.noreply.github.com> Co-authored-by: Aidan Daly <aidandal@amazon.com> Co-authored-by: Harrison Weinstock <hkobew@amazon.com> Co-authored-by: aws-aditya21 <saivenki@amazon.com> Co-authored-by: notgitika <gitijh@gmail.com>

notgitika · 2026-05-04T21:33:27Z

Summary

Adds missing IAM permissions to docs policy files for batch evaluation and recommendations:

Add BatchEvalAndRecommendations statement to iam-policy-user.json (9 actions)
Add CloudWatch Logs write permissions to LogsStreamingAndSearch (DescribeLogGroups, CreateLogGroup, CreateLogStream, PutLogEvents, PutRetentionPolicy)
Update PERMISSIONS.md with batch eval/recommendations reference section and logging entries

Also applied logs:PutRetentionPolicy to the e2e-github-actions role in the test account.

Context

E2E batch eval tests were failing with 403 on PutRetentionPolicy -- the batch eval API sets a retention policy on its results log group.

Test plan

iam-policy-user.json is valid JSON
prettier passes
Applied PutRetentionPolicy to e2e-github-actions role
Re-run E2E to confirm batch eval tests pass

…sions to docs

agentcore-cli-automation · 2026-05-04T21:36:53Z

Thanks for adding these permissions. One issue worth addressing before merge:

The CLI Commands column in PERMISSIONS.md references commands that don't exist in the CLI today.

In the new "Batch evaluation and recommendations" table:

run batch-eval — no such command. run only has the eval subcommand (see src/cli/commands/run/command.tsx).
run recommend — no such command.
evals history is listed for bedrock-agentcore:ListBatchEvaluations, but evals history is a local-only command that reads from disk via handleListEvalRuns → listEvalRuns() (see src/cli/operations/eval/list-eval-runs.ts and src/cli/commands/eval/command.tsx). It makes no AWS API call, so it doesn't require ListBatchEvaluations.

I also couldn't find any references to StartBatchEvaluation, StartRecommendation, etc. anywhere in src/ or the e2e tests — the run eval code path uses the Evaluate API (already covered).

If the service is invoking batch eval / recommendation APIs server-side on behalf of the user when run eval is called (which would explain the PutRetentionPolicy 403 you mentioned in the description), it might be worth:

Options to address this:

Update the CLI Commands column entries to point to the actual command that triggers these permissions (e.g. run evals if that's the command that indirectly causes batch eval invocations server-side), and update the Purpose column to explain it's service-invoked on behalf of the caller.
Drop the Batch evaluation and recommendations section from PERMISSIONS.md and the BatchEvalAndRecommendations statement from iam-policy-user.json for now if no CLI command actually calls these APIs — and only keep the logs permissions that are actually failing (PutRetentionPolicy, etc.).
If these commands are planned/upcoming, either add them in this PR or gate the doc changes until the commands land, so the docs don't advertise commands users can't run.

The logs:* additions (PutRetentionPolicy, CreateLogGroup, CreateLogStream, PutLogEvents, DescribeLogGroups) look fine as long as the batch-eval-via-service-principal explanation holds, though the CLI Commands column still lists the non-existent run batch-eval for most of them and should be fixed the same way.

github-actions · 2026-05-04T21:37:13Z

Coverage Report

Status	Category	Percentage	Covered / Total
🔵	Lines	43.04%	8990 / 20883
🔵	Statements	42.33%	9546 / 22550
🔵	Functions	39.91%	1552 / 3888
🔵	Branches	39.92%	5794 / 14511

Generated in workflow #2378 for commit 1b9ef28 by the Vitest Coverage Report Action

Cherry-picks non-breaking changes from main: - docs: add telemetry instrumentation guide (#1197) - chore: replace PAT tokens with GitHub App token (#1198) - fix: add batch eval, recommendation, CloudWatch Logs permissions (#1113) - feat(evaluator): add kmsKeyArn support (#994) - chore: replace github.token with GitHub App token (#1210) - fix: sync-preview workflow rewrite (#1078) Skipped (requires dedicated effort due to Result type refactor): - refactor: unify result types with Result<T, E> (#1125) - feat: instrument telemetry for create/deploy (#1202, #1206) - feat: record command attrs on failure (#1204)

* docs: add telemetry instrumentation guide (#1197) * chore: replace PAT tokens with GitHub App token (#1198) Replace secrets.PAT_TOKEN and secrets.AUTOMATION_ACCOUNT_PAT_TOKEN with short-lived tokens generated by the agentcore-devx-automation GitHub App (ID: 3637953) via actions/create-github-app-token@v1. This improves security by using ephemeral tokens scoped to the installation rather than long-lived personal access tokens. Requires adding repo variable APP_ID=3637953 and repo secret APP_PRIVATE_KEY with the app's RSA private key. * fix: add batch eval, recommendation, and CloudWatch Logs write permissions to docs (#1113) * feat: instrument telemetry for create command (CLI + TUI) (#1202) Add telemetry recording to the create command in both CLI and TUI paths: - CLI: wrap handleCreateCLI with runCliCommand to emit CreateAttrs on success/failure - TUI: wrap useCreateFlow's run() with withCommandRunTelemetry - Add telemetry assertions to existing integration tests (frameworks + edge cases) * chore: replace all github.token/GITHUB_TOKEN with GitHub App token Replaces all occurrences of \${{ github.token }} and \${{ secrets.GITHUB_TOKEN }} across .github/workflows/ with a per-job GitHub App token generated via actions/create-github-app-token@v1 using vars.APP_ID and secrets.APP_PRIVATE_KEY. * fix: bump versions to resolve security audit failure * revert: keep pr-title.yml using GITHUB_TOKEN (read-only access sufficient) * feat(evaluator): add kmsKeyArn support for custom evaluator (#994) * feat(evaluator): Add kmsKeyArn support for custom evaluator * fix: sync package-lock.json with package.json The lock file was out of sync after dependency bumps on main were merged, causing npm ci to fail in CI. * fix: revert unrelated dep bumps and fix formatting Reverts @opentelemetry/exporter-metrics-otlp-http ^0.217.0 back to ^0.214.0 and secretlint ^13.0.0 back to ^12.2.0 — these were accidentally included in the feature commit from unmerged dependabot PRs and introduce high-severity protobufjs vulnerabilities. Restores fast-xml-parser and @aws-sdk/xml-builder overrides that were also inadvertently removed. Fixes Prettier formatting on agentcore-project.ts import lines. * fix: sync package-lock.json with updated dependencies --------- Co-authored-by: notgitika <gitijh@gmail.com> * refactor: unify result types with discriminated Result<T, E> union (#1125) * refactor: unify result types with discriminated Result<T, E> union Introduce a shared Result<T, E> type (inspired by Rust's Result) that replaces ad-hoc { success: boolean; error?: string } patterns across the codebase. Key changes: - Add src/lib/types.ts with Result<T, E> discriminated union type - Add toError() helper in src/cli/errors.ts for catch blocks - Migrate all command, operation, and primitive result types to Result<T> - Error field is now Error (not string) on the failure branch - Data fields only exist on the success branch (proper narrowing) - Update all consumers to narrow before accessing branch-specific fields - Update test assertions to match new Error objects and add narrowing * docs: update AGENTS.md and telemetry README to reflect Result<T, E> type * feat: record command attrs on telemetry failure via fallbackAttrs (#1204) * feat: record command attrs on telemetry failure via fallbackAttrs Add optional fallbackAttrs parameter to client.withCommandRun so command-specific attributes are recorded even when the callback throws. - client.ts: accept fallbackAttrs, use on failure instead of {} - client.ts: run resilientParse on all non-empty attrs (not just success) - cli-command-run.ts: withCommandRunTelemetry passes attrs as fallbackAttrs - cli-command-run.ts: runCliCommand accepts optional knownAttrs param - command.tsx: extract knownAttrs upfront, pass to runCliCommand - client.test.ts: add unit tests for fallbackAttrs behavior - create-edge-cases.test.ts: assert attrs present on failure entry * chore: rebase onto mainline * fix: sync-preview pushes directly on clean merge, PRs only on conflict (#1078) * fix: sync-preview workflow restores version instead of ignoring files Instead of keeping preview's entire package.json/package-lock.json (which discards new deps, scripts, etc. from main), accept main's content and surgically restore only the version field to preview's value after merge. * fix: push directly to preview on clean merge via GitHub App bypass Use agentcore-devx-automation app token to bypass branch protection and push directly when the merge is clean (or only version conflicts). Only creates a PR when there are real conflicts in other files. * chore: use app-slug instead of app-id for token generation * fix: address review feedback on sync-preview workflow - Pass PREVIEW_VERSION via env var instead of string interpolation in node -e scripts (safer against special chars) - Make git add of package-lock.json conditional on file existence to match the earlier -f guard - Replace loose title search for dedup with headRefName prefix filter to avoid false positives from unrelated PRs - Clarify why package.json/package-lock.json are special-cased (preview carries a different version string that needs preserving) * fix: restore preview-owned files after sync merge Adds a step to restore schemas/agentcore.schema.v1.json and CHANGELOG.md to preview's versions after merging main. These files are auto-generated during preview releases — schema-check CI rejects direct modifications to schemas/, and CHANGELOG.md tracks preview releases separately. * fix: use app-id instead of app-slug for GitHub App token Aligns with the pattern in PR #1210 and ci-failure-issue.yml. * feat: instrument telemetry for deploy command (CLI + TUI) (#1206) * chore: sync safe commits from main into preview Cherry-picks non-breaking changes from main: - docs: add telemetry instrumentation guide (#1197) - chore: replace PAT tokens with GitHub App token (#1198) - fix: add batch eval, recommendation, CloudWatch Logs permissions (#1113) - feat(evaluator): add kmsKeyArn support (#994) - chore: replace github.token with GitHub App token (#1210) - fix: sync-preview workflow rewrite (#1078) Skipped (requires dedicated effort due to Result type refactor): - refactor: unify result types with Result<T, E> (#1125) - feat: instrument telemetry for create/deploy (#1202, #1206) - feat: record command attrs on failure (#1204) * chore: merge main into preview with Result refactor Brings in all remaining main commits including: - refactor: unify result types with Result<T, E> (#1125) - feat: instrument telemetry for deploy command (#1206) - feat: record command attrs on failure (#1204) - feat: instrument telemetry for create command (#1202) Adapts preview's deploy flow to use OperationResult with string errors for compatibility with the telemetry layer. --------- Co-authored-by: Hweinstock <42325418+Hweinstock@users.noreply.github.com> Co-authored-by: Aidan Daly <99039782+aidandaly24@users.noreply.github.com> Co-authored-by: Gitika <53349492+notgitika@users.noreply.github.com> Co-authored-by: Aidan Daly <aidandal@amazon.com> Co-authored-by: Harrison Weinstock <hkobew@amazon.com> Co-authored-by: aws-aditya21 <saivenki@amazon.com> Co-authored-by: notgitika <gitijh@gmail.com>

notgitika · 2026-05-04T21:33:27Z

Summary

Adds missing IAM permissions to docs policy files for batch evaluation and recommendations:

Add BatchEvalAndRecommendations statement to iam-policy-user.json (9 actions)
Add CloudWatch Logs write permissions to LogsStreamingAndSearch (DescribeLogGroups, CreateLogGroup, CreateLogStream, PutLogEvents, PutRetentionPolicy)
Update PERMISSIONS.md with batch eval/recommendations reference section and logging entries

Also applied logs:PutRetentionPolicy to the e2e-github-actions role in the test account.

Context

E2E batch eval tests were failing with 403 on PutRetentionPolicy -- the batch eval API sets a retention policy on its results log group.

Test plan

iam-policy-user.json is valid JSON
prettier passes
Applied PutRetentionPolicy to e2e-github-actions role
Re-run E2E to confirm batch eval tests pass

…sions to docs

agentcore-cli-automation · 2026-05-04T21:36:53Z

Thanks for adding these permissions. One issue worth addressing before merge:

The CLI Commands column in PERMISSIONS.md references commands that don't exist in the CLI today.

In the new "Batch evaluation and recommendations" table:

run batch-eval — no such command. run only has the eval subcommand (see src/cli/commands/run/command.tsx).
run recommend — no such command.
evals history is listed for bedrock-agentcore:ListBatchEvaluations, but evals history is a local-only command that reads from disk via handleListEvalRuns → listEvalRuns() (see src/cli/operations/eval/list-eval-runs.ts and src/cli/commands/eval/command.tsx). It makes no AWS API call, so it doesn't require ListBatchEvaluations.

I also couldn't find any references to StartBatchEvaluation, StartRecommendation, etc. anywhere in src/ or the e2e tests — the run eval code path uses the Evaluate API (already covered).

If the service is invoking batch eval / recommendation APIs server-side on behalf of the user when run eval is called (which would explain the PutRetentionPolicy 403 you mentioned in the description), it might be worth:

Options to address this:

Update the CLI Commands column entries to point to the actual command that triggers these permissions (e.g. run evals if that's the command that indirectly causes batch eval invocations server-side), and update the Purpose column to explain it's service-invoked on behalf of the caller.
Drop the Batch evaluation and recommendations section from PERMISSIONS.md and the BatchEvalAndRecommendations statement from iam-policy-user.json for now if no CLI command actually calls these APIs — and only keep the logs permissions that are actually failing (PutRetentionPolicy, etc.).
If these commands are planned/upcoming, either add them in this PR or gate the doc changes until the commands land, so the docs don't advertise commands users can't run.

The logs:* additions (PutRetentionPolicy, CreateLogGroup, CreateLogStream, PutLogEvents, DescribeLogGroups) look fine as long as the batch-eval-via-service-principal explanation holds, though the CLI Commands column still lists the non-existent run batch-eval for most of them and should be fixed the same way.

github-actions · 2026-05-04T21:37:13Z

Coverage Report

Status	Category	Percentage	Covered / Total
🔵	Lines	43.04%	8990 / 20883
🔵	Statements	42.33%	9546 / 22550
🔵	Functions	39.91%	1552 / 3888
🔵	Branches	39.92%	5794 / 14511

Generated in workflow #2378 for commit 1b9ef28 by the Vitest Coverage Report Action

Cherry-picks non-breaking changes from main: - docs: add telemetry instrumentation guide (#1197) - chore: replace PAT tokens with GitHub App token (#1198) - fix: add batch eval, recommendation, CloudWatch Logs permissions (#1113) - feat(evaluator): add kmsKeyArn support (#994) - chore: replace github.token with GitHub App token (#1210) - fix: sync-preview workflow rewrite (#1078) Skipped (requires dedicated effort due to Result type refactor): - refactor: unify result types with Result<T, E> (#1125) - feat: instrument telemetry for create/deploy (#1202, #1206) - feat: record command attrs on failure (#1204)

* docs: add telemetry instrumentation guide (#1197) * chore: replace PAT tokens with GitHub App token (#1198) Replace secrets.PAT_TOKEN and secrets.AUTOMATION_ACCOUNT_PAT_TOKEN with short-lived tokens generated by the agentcore-devx-automation GitHub App (ID: 3637953) via actions/create-github-app-token@v1. This improves security by using ephemeral tokens scoped to the installation rather than long-lived personal access tokens. Requires adding repo variable APP_ID=3637953 and repo secret APP_PRIVATE_KEY with the app's RSA private key. * fix: add batch eval, recommendation, and CloudWatch Logs write permissions to docs (#1113) * feat: instrument telemetry for create command (CLI + TUI) (#1202) Add telemetry recording to the create command in both CLI and TUI paths: - CLI: wrap handleCreateCLI with runCliCommand to emit CreateAttrs on success/failure - TUI: wrap useCreateFlow's run() with withCommandRunTelemetry - Add telemetry assertions to existing integration tests (frameworks + edge cases) * chore: replace all github.token/GITHUB_TOKEN with GitHub App token Replaces all occurrences of \${{ github.token }} and \${{ secrets.GITHUB_TOKEN }} across .github/workflows/ with a per-job GitHub App token generated via actions/create-github-app-token@v1 using vars.APP_ID and secrets.APP_PRIVATE_KEY. * fix: bump versions to resolve security audit failure * revert: keep pr-title.yml using GITHUB_TOKEN (read-only access sufficient) * feat(evaluator): add kmsKeyArn support for custom evaluator (#994) * feat(evaluator): Add kmsKeyArn support for custom evaluator * fix: sync package-lock.json with package.json The lock file was out of sync after dependency bumps on main were merged, causing npm ci to fail in CI. * fix: revert unrelated dep bumps and fix formatting Reverts @opentelemetry/exporter-metrics-otlp-http ^0.217.0 back to ^0.214.0 and secretlint ^13.0.0 back to ^12.2.0 — these were accidentally included in the feature commit from unmerged dependabot PRs and introduce high-severity protobufjs vulnerabilities. Restores fast-xml-parser and @aws-sdk/xml-builder overrides that were also inadvertently removed. Fixes Prettier formatting on agentcore-project.ts import lines. * fix: sync package-lock.json with updated dependencies --------- Co-authored-by: notgitika <gitijh@gmail.com> * refactor: unify result types with discriminated Result<T, E> union (#1125) * refactor: unify result types with discriminated Result<T, E> union Introduce a shared Result<T, E> type (inspired by Rust's Result) that replaces ad-hoc { success: boolean; error?: string } patterns across the codebase. Key changes: - Add src/lib/types.ts with Result<T, E> discriminated union type - Add toError() helper in src/cli/errors.ts for catch blocks - Migrate all command, operation, and primitive result types to Result<T> - Error field is now Error (not string) on the failure branch - Data fields only exist on the success branch (proper narrowing) - Update all consumers to narrow before accessing branch-specific fields - Update test assertions to match new Error objects and add narrowing * docs: update AGENTS.md and telemetry README to reflect Result<T, E> type * feat: record command attrs on telemetry failure via fallbackAttrs (#1204) * feat: record command attrs on telemetry failure via fallbackAttrs Add optional fallbackAttrs parameter to client.withCommandRun so command-specific attributes are recorded even when the callback throws. - client.ts: accept fallbackAttrs, use on failure instead of {} - client.ts: run resilientParse on all non-empty attrs (not just success) - cli-command-run.ts: withCommandRunTelemetry passes attrs as fallbackAttrs - cli-command-run.ts: runCliCommand accepts optional knownAttrs param - command.tsx: extract knownAttrs upfront, pass to runCliCommand - client.test.ts: add unit tests for fallbackAttrs behavior - create-edge-cases.test.ts: assert attrs present on failure entry * chore: rebase onto mainline * fix: sync-preview pushes directly on clean merge, PRs only on conflict (#1078) * fix: sync-preview workflow restores version instead of ignoring files Instead of keeping preview's entire package.json/package-lock.json (which discards new deps, scripts, etc. from main), accept main's content and surgically restore only the version field to preview's value after merge. * fix: push directly to preview on clean merge via GitHub App bypass Use agentcore-devx-automation app token to bypass branch protection and push directly when the merge is clean (or only version conflicts). Only creates a PR when there are real conflicts in other files. * chore: use app-slug instead of app-id for token generation * fix: address review feedback on sync-preview workflow - Pass PREVIEW_VERSION via env var instead of string interpolation in node -e scripts (safer against special chars) - Make git add of package-lock.json conditional on file existence to match the earlier -f guard - Replace loose title search for dedup with headRefName prefix filter to avoid false positives from unrelated PRs - Clarify why package.json/package-lock.json are special-cased (preview carries a different version string that needs preserving) * fix: restore preview-owned files after sync merge Adds a step to restore schemas/agentcore.schema.v1.json and CHANGELOG.md to preview's versions after merging main. These files are auto-generated during preview releases — schema-check CI rejects direct modifications to schemas/, and CHANGELOG.md tracks preview releases separately. * fix: use app-id instead of app-slug for GitHub App token Aligns with the pattern in PR #1210 and ci-failure-issue.yml. * feat: instrument telemetry for deploy command (CLI + TUI) (#1206) * chore: sync safe commits from main into preview Cherry-picks non-breaking changes from main: - docs: add telemetry instrumentation guide (#1197) - chore: replace PAT tokens with GitHub App token (#1198) - fix: add batch eval, recommendation, CloudWatch Logs permissions (#1113) - feat(evaluator): add kmsKeyArn support (#994) - chore: replace github.token with GitHub App token (#1210) - fix: sync-preview workflow rewrite (#1078) Skipped (requires dedicated effort due to Result type refactor): - refactor: unify result types with Result<T, E> (#1125) - feat: instrument telemetry for create/deploy (#1202, #1206) - feat: record command attrs on failure (#1204) * chore: merge main into preview with Result refactor Brings in all remaining main commits including: - refactor: unify result types with Result<T, E> (#1125) - feat: instrument telemetry for deploy command (#1206) - feat: record command attrs on failure (#1204) - feat: instrument telemetry for create command (#1202) Adapts preview's deploy flow to use OperationResult with string errors for compatibility with the telemetry layer. --------- Co-authored-by: Hweinstock <42325418+Hweinstock@users.noreply.github.com> Co-authored-by: Aidan Daly <99039782+aidandaly24@users.noreply.github.com> Co-authored-by: Gitika <53349492+notgitika@users.noreply.github.com> Co-authored-by: Aidan Daly <aidandal@amazon.com> Co-authored-by: Harrison Weinstock <hkobew@amazon.com> Co-authored-by: aws-aditya21 <saivenki@amazon.com> Co-authored-by: notgitika <gitijh@gmail.com>

notgitika · 2026-05-04T21:33:27Z

Summary

Adds missing IAM permissions to docs policy files for batch evaluation and recommendations:

Add BatchEvalAndRecommendations statement to iam-policy-user.json (9 actions)
Add CloudWatch Logs write permissions to LogsStreamingAndSearch (DescribeLogGroups, CreateLogGroup, CreateLogStream, PutLogEvents, PutRetentionPolicy)
Update PERMISSIONS.md with batch eval/recommendations reference section and logging entries

Also applied logs:PutRetentionPolicy to the e2e-github-actions role in the test account.

Context

E2E batch eval tests were failing with 403 on PutRetentionPolicy -- the batch eval API sets a retention policy on its results log group.

Test plan

iam-policy-user.json is valid JSON
prettier passes
Applied PutRetentionPolicy to e2e-github-actions role
Re-run E2E to confirm batch eval tests pass

…sions to docs

agentcore-cli-automation · 2026-05-04T21:36:53Z

Thanks for adding these permissions. One issue worth addressing before merge:

The CLI Commands column in PERMISSIONS.md references commands that don't exist in the CLI today.

In the new "Batch evaluation and recommendations" table:

run batch-eval — no such command. run only has the eval subcommand (see src/cli/commands/run/command.tsx).
run recommend — no such command.
evals history is listed for bedrock-agentcore:ListBatchEvaluations, but evals history is a local-only command that reads from disk via handleListEvalRuns → listEvalRuns() (see src/cli/operations/eval/list-eval-runs.ts and src/cli/commands/eval/command.tsx). It makes no AWS API call, so it doesn't require ListBatchEvaluations.

I also couldn't find any references to StartBatchEvaluation, StartRecommendation, etc. anywhere in src/ or the e2e tests — the run eval code path uses the Evaluate API (already covered).

If the service is invoking batch eval / recommendation APIs server-side on behalf of the user when run eval is called (which would explain the PutRetentionPolicy 403 you mentioned in the description), it might be worth:

Options to address this:

Update the CLI Commands column entries to point to the actual command that triggers these permissions (e.g. run evals if that's the command that indirectly causes batch eval invocations server-side), and update the Purpose column to explain it's service-invoked on behalf of the caller.
Drop the Batch evaluation and recommendations section from PERMISSIONS.md and the BatchEvalAndRecommendations statement from iam-policy-user.json for now if no CLI command actually calls these APIs — and only keep the logs permissions that are actually failing (PutRetentionPolicy, etc.).
If these commands are planned/upcoming, either add them in this PR or gate the doc changes until the commands land, so the docs don't advertise commands users can't run.

The logs:* additions (PutRetentionPolicy, CreateLogGroup, CreateLogStream, PutLogEvents, DescribeLogGroups) look fine as long as the batch-eval-via-service-principal explanation holds, though the CLI Commands column still lists the non-existent run batch-eval for most of them and should be fixed the same way.

github-actions · 2026-05-04T21:37:13Z

Coverage Report

Status	Category	Percentage	Covered / Total
🔵	Lines	43.04%	8990 / 20883
🔵	Statements	42.33%	9546 / 22550
🔵	Functions	39.91%	1552 / 3888
🔵	Branches	39.92%	5794 / 14511

Generated in workflow #2378 for commit 1b9ef28 by the Vitest Coverage Report Action

Cherry-picks non-breaking changes from main: - docs: add telemetry instrumentation guide (#1197) - chore: replace PAT tokens with GitHub App token (#1198) - fix: add batch eval, recommendation, CloudWatch Logs permissions (#1113) - feat(evaluator): add kmsKeyArn support (#994) - chore: replace github.token with GitHub App token (#1210) - fix: sync-preview workflow rewrite (#1078) Skipped (requires dedicated effort due to Result type refactor): - refactor: unify result types with Result<T, E> (#1125) - feat: instrument telemetry for create/deploy (#1202, #1206) - feat: record command attrs on failure (#1204)

* docs: add telemetry instrumentation guide (#1197) * chore: replace PAT tokens with GitHub App token (#1198) Replace secrets.PAT_TOKEN and secrets.AUTOMATION_ACCOUNT_PAT_TOKEN with short-lived tokens generated by the agentcore-devx-automation GitHub App (ID: 3637953) via actions/create-github-app-token@v1. This improves security by using ephemeral tokens scoped to the installation rather than long-lived personal access tokens. Requires adding repo variable APP_ID=3637953 and repo secret APP_PRIVATE_KEY with the app's RSA private key. * fix: add batch eval, recommendation, and CloudWatch Logs write permissions to docs (#1113) * feat: instrument telemetry for create command (CLI + TUI) (#1202) Add telemetry recording to the create command in both CLI and TUI paths: - CLI: wrap handleCreateCLI with runCliCommand to emit CreateAttrs on success/failure - TUI: wrap useCreateFlow's run() with withCommandRunTelemetry - Add telemetry assertions to existing integration tests (frameworks + edge cases) * chore: replace all github.token/GITHUB_TOKEN with GitHub App token Replaces all occurrences of \${{ github.token }} and \${{ secrets.GITHUB_TOKEN }} across .github/workflows/ with a per-job GitHub App token generated via actions/create-github-app-token@v1 using vars.APP_ID and secrets.APP_PRIVATE_KEY. * fix: bump versions to resolve security audit failure * revert: keep pr-title.yml using GITHUB_TOKEN (read-only access sufficient) * feat(evaluator): add kmsKeyArn support for custom evaluator (#994) * feat(evaluator): Add kmsKeyArn support for custom evaluator * fix: sync package-lock.json with package.json The lock file was out of sync after dependency bumps on main were merged, causing npm ci to fail in CI. * fix: revert unrelated dep bumps and fix formatting Reverts @opentelemetry/exporter-metrics-otlp-http ^0.217.0 back to ^0.214.0 and secretlint ^13.0.0 back to ^12.2.0 — these were accidentally included in the feature commit from unmerged dependabot PRs and introduce high-severity protobufjs vulnerabilities. Restores fast-xml-parser and @aws-sdk/xml-builder overrides that were also inadvertently removed. Fixes Prettier formatting on agentcore-project.ts import lines. * fix: sync package-lock.json with updated dependencies --------- Co-authored-by: notgitika <gitijh@gmail.com> * refactor: unify result types with discriminated Result<T, E> union (#1125) * refactor: unify result types with discriminated Result<T, E> union Introduce a shared Result<T, E> type (inspired by Rust's Result) that replaces ad-hoc { success: boolean; error?: string } patterns across the codebase. Key changes: - Add src/lib/types.ts with Result<T, E> discriminated union type - Add toError() helper in src/cli/errors.ts for catch blocks - Migrate all command, operation, and primitive result types to Result<T> - Error field is now Error (not string) on the failure branch - Data fields only exist on the success branch (proper narrowing) - Update all consumers to narrow before accessing branch-specific fields - Update test assertions to match new Error objects and add narrowing * docs: update AGENTS.md and telemetry README to reflect Result<T, E> type * feat: record command attrs on telemetry failure via fallbackAttrs (#1204) * feat: record command attrs on telemetry failure via fallbackAttrs Add optional fallbackAttrs parameter to client.withCommandRun so command-specific attributes are recorded even when the callback throws. - client.ts: accept fallbackAttrs, use on failure instead of {} - client.ts: run resilientParse on all non-empty attrs (not just success) - cli-command-run.ts: withCommandRunTelemetry passes attrs as fallbackAttrs - cli-command-run.ts: runCliCommand accepts optional knownAttrs param - command.tsx: extract knownAttrs upfront, pass to runCliCommand - client.test.ts: add unit tests for fallbackAttrs behavior - create-edge-cases.test.ts: assert attrs present on failure entry * chore: rebase onto mainline * fix: sync-preview pushes directly on clean merge, PRs only on conflict (#1078) * fix: sync-preview workflow restores version instead of ignoring files Instead of keeping preview's entire package.json/package-lock.json (which discards new deps, scripts, etc. from main), accept main's content and surgically restore only the version field to preview's value after merge. * fix: push directly to preview on clean merge via GitHub App bypass Use agentcore-devx-automation app token to bypass branch protection and push directly when the merge is clean (or only version conflicts). Only creates a PR when there are real conflicts in other files. * chore: use app-slug instead of app-id for token generation * fix: address review feedback on sync-preview workflow - Pass PREVIEW_VERSION via env var instead of string interpolation in node -e scripts (safer against special chars) - Make git add of package-lock.json conditional on file existence to match the earlier -f guard - Replace loose title search for dedup with headRefName prefix filter to avoid false positives from unrelated PRs - Clarify why package.json/package-lock.json are special-cased (preview carries a different version string that needs preserving) * fix: restore preview-owned files after sync merge Adds a step to restore schemas/agentcore.schema.v1.json and CHANGELOG.md to preview's versions after merging main. These files are auto-generated during preview releases — schema-check CI rejects direct modifications to schemas/, and CHANGELOG.md tracks preview releases separately. * fix: use app-id instead of app-slug for GitHub App token Aligns with the pattern in PR #1210 and ci-failure-issue.yml. * feat: instrument telemetry for deploy command (CLI + TUI) (#1206) * chore: sync safe commits from main into preview Cherry-picks non-breaking changes from main: - docs: add telemetry instrumentation guide (#1197) - chore: replace PAT tokens with GitHub App token (#1198) - fix: add batch eval, recommendation, CloudWatch Logs permissions (#1113) - feat(evaluator): add kmsKeyArn support (#994) - chore: replace github.token with GitHub App token (#1210) - fix: sync-preview workflow rewrite (#1078) Skipped (requires dedicated effort due to Result type refactor): - refactor: unify result types with Result<T, E> (#1125) - feat: instrument telemetry for create/deploy (#1202, #1206) - feat: record command attrs on failure (#1204) * chore: merge main into preview with Result refactor Brings in all remaining main commits including: - refactor: unify result types with Result<T, E> (#1125) - feat: instrument telemetry for deploy command (#1206) - feat: record command attrs on failure (#1204) - feat: instrument telemetry for create command (#1202) Adapts preview's deploy flow to use OperationResult with string errors for compatibility with the telemetry layer. --------- Co-authored-by: Hweinstock <42325418+Hweinstock@users.noreply.github.com> Co-authored-by: Aidan Daly <99039782+aidandaly24@users.noreply.github.com> Co-authored-by: Gitika <53349492+notgitika@users.noreply.github.com> Co-authored-by: Aidan Daly <aidandal@amazon.com> Co-authored-by: Harrison Weinstock <hkobew@amazon.com> Co-authored-by: aws-aditya21 <saivenki@amazon.com> Co-authored-by: notgitika <gitijh@gmail.com>

notgitika · 2026-05-04T21:33:27Z

Summary

Adds missing IAM permissions to docs policy files for batch evaluation and recommendations:

Add BatchEvalAndRecommendations statement to iam-policy-user.json (9 actions)
Add CloudWatch Logs write permissions to LogsStreamingAndSearch (DescribeLogGroups, CreateLogGroup, CreateLogStream, PutLogEvents, PutRetentionPolicy)
Update PERMISSIONS.md with batch eval/recommendations reference section and logging entries

Also applied logs:PutRetentionPolicy to the e2e-github-actions role in the test account.

Context

E2E batch eval tests were failing with 403 on PutRetentionPolicy -- the batch eval API sets a retention policy on its results log group.

Test plan

iam-policy-user.json is valid JSON
prettier passes
Applied PutRetentionPolicy to e2e-github-actions role
Re-run E2E to confirm batch eval tests pass

…sions to docs

agentcore-cli-automation · 2026-05-04T21:36:53Z

Thanks for adding these permissions. One issue worth addressing before merge:

The CLI Commands column in PERMISSIONS.md references commands that don't exist in the CLI today.

In the new "Batch evaluation and recommendations" table:

run batch-eval — no such command. run only has the eval subcommand (see src/cli/commands/run/command.tsx).
run recommend — no such command.
evals history is listed for bedrock-agentcore:ListBatchEvaluations, but evals history is a local-only command that reads from disk via handleListEvalRuns → listEvalRuns() (see src/cli/operations/eval/list-eval-runs.ts and src/cli/commands/eval/command.tsx). It makes no AWS API call, so it doesn't require ListBatchEvaluations.

I also couldn't find any references to StartBatchEvaluation, StartRecommendation, etc. anywhere in src/ or the e2e tests — the run eval code path uses the Evaluate API (already covered).

If the service is invoking batch eval / recommendation APIs server-side on behalf of the user when run eval is called (which would explain the PutRetentionPolicy 403 you mentioned in the description), it might be worth:

Options to address this:

Update the CLI Commands column entries to point to the actual command that triggers these permissions (e.g. run evals if that's the command that indirectly causes batch eval invocations server-side), and update the Purpose column to explain it's service-invoked on behalf of the caller.
Drop the Batch evaluation and recommendations section from PERMISSIONS.md and the BatchEvalAndRecommendations statement from iam-policy-user.json for now if no CLI command actually calls these APIs — and only keep the logs permissions that are actually failing (PutRetentionPolicy, etc.).
If these commands are planned/upcoming, either add them in this PR or gate the doc changes until the commands land, so the docs don't advertise commands users can't run.

The logs:* additions (PutRetentionPolicy, CreateLogGroup, CreateLogStream, PutLogEvents, DescribeLogGroups) look fine as long as the batch-eval-via-service-principal explanation holds, though the CLI Commands column still lists the non-existent run batch-eval for most of them and should be fixed the same way.

github-actions · 2026-05-04T21:37:13Z

Coverage Report

Status	Category	Percentage	Covered / Total
🔵	Lines	43.04%	8990 / 20883
🔵	Statements	42.33%	9546 / 22550
🔵	Functions	39.91%	1552 / 3888
🔵	Branches	39.92%	5794 / 14511

Generated in workflow #2378 for commit 1b9ef28 by the Vitest Coverage Report Action

Cherry-picks non-breaking changes from main: - docs: add telemetry instrumentation guide (#1197) - chore: replace PAT tokens with GitHub App token (#1198) - fix: add batch eval, recommendation, CloudWatch Logs permissions (#1113) - feat(evaluator): add kmsKeyArn support (#994) - chore: replace github.token with GitHub App token (#1210) - fix: sync-preview workflow rewrite (#1078) Skipped (requires dedicated effort due to Result type refactor): - refactor: unify result types with Result<T, E> (#1125) - feat: instrument telemetry for create/deploy (#1202, #1206) - feat: record command attrs on failure (#1204)

* docs: add telemetry instrumentation guide (#1197) * chore: replace PAT tokens with GitHub App token (#1198) Replace secrets.PAT_TOKEN and secrets.AUTOMATION_ACCOUNT_PAT_TOKEN with short-lived tokens generated by the agentcore-devx-automation GitHub App (ID: 3637953) via actions/create-github-app-token@v1. This improves security by using ephemeral tokens scoped to the installation rather than long-lived personal access tokens. Requires adding repo variable APP_ID=3637953 and repo secret APP_PRIVATE_KEY with the app's RSA private key. * fix: add batch eval, recommendation, and CloudWatch Logs write permissions to docs (#1113) * feat: instrument telemetry for create command (CLI + TUI) (#1202) Add telemetry recording to the create command in both CLI and TUI paths: - CLI: wrap handleCreateCLI with runCliCommand to emit CreateAttrs on success/failure - TUI: wrap useCreateFlow's run() with withCommandRunTelemetry - Add telemetry assertions to existing integration tests (frameworks + edge cases) * chore: replace all github.token/GITHUB_TOKEN with GitHub App token Replaces all occurrences of \${{ github.token }} and \${{ secrets.GITHUB_TOKEN }} across .github/workflows/ with a per-job GitHub App token generated via actions/create-github-app-token@v1 using vars.APP_ID and secrets.APP_PRIVATE_KEY. * fix: bump versions to resolve security audit failure * revert: keep pr-title.yml using GITHUB_TOKEN (read-only access sufficient) * feat(evaluator): add kmsKeyArn support for custom evaluator (#994) * feat(evaluator): Add kmsKeyArn support for custom evaluator * fix: sync package-lock.json with package.json The lock file was out of sync after dependency bumps on main were merged, causing npm ci to fail in CI. * fix: revert unrelated dep bumps and fix formatting Reverts @opentelemetry/exporter-metrics-otlp-http ^0.217.0 back to ^0.214.0 and secretlint ^13.0.0 back to ^12.2.0 — these were accidentally included in the feature commit from unmerged dependabot PRs and introduce high-severity protobufjs vulnerabilities. Restores fast-xml-parser and @aws-sdk/xml-builder overrides that were also inadvertently removed. Fixes Prettier formatting on agentcore-project.ts import lines. * fix: sync package-lock.json with updated dependencies --------- Co-authored-by: notgitika <gitijh@gmail.com> * refactor: unify result types with discriminated Result<T, E> union (#1125) * refactor: unify result types with discriminated Result<T, E> union Introduce a shared Result<T, E> type (inspired by Rust's Result) that replaces ad-hoc { success: boolean; error?: string } patterns across the codebase. Key changes: - Add src/lib/types.ts with Result<T, E> discriminated union type - Add toError() helper in src/cli/errors.ts for catch blocks - Migrate all command, operation, and primitive result types to Result<T> - Error field is now Error (not string) on the failure branch - Data fields only exist on the success branch (proper narrowing) - Update all consumers to narrow before accessing branch-specific fields - Update test assertions to match new Error objects and add narrowing * docs: update AGENTS.md and telemetry README to reflect Result<T, E> type * feat: record command attrs on telemetry failure via fallbackAttrs (#1204) * feat: record command attrs on telemetry failure via fallbackAttrs Add optional fallbackAttrs parameter to client.withCommandRun so command-specific attributes are recorded even when the callback throws. - client.ts: accept fallbackAttrs, use on failure instead of {} - client.ts: run resilientParse on all non-empty attrs (not just success) - cli-command-run.ts: withCommandRunTelemetry passes attrs as fallbackAttrs - cli-command-run.ts: runCliCommand accepts optional knownAttrs param - command.tsx: extract knownAttrs upfront, pass to runCliCommand - client.test.ts: add unit tests for fallbackAttrs behavior - create-edge-cases.test.ts: assert attrs present on failure entry * chore: rebase onto mainline * fix: sync-preview pushes directly on clean merge, PRs only on conflict (#1078) * fix: sync-preview workflow restores version instead of ignoring files Instead of keeping preview's entire package.json/package-lock.json (which discards new deps, scripts, etc. from main), accept main's content and surgically restore only the version field to preview's value after merge. * fix: push directly to preview on clean merge via GitHub App bypass Use agentcore-devx-automation app token to bypass branch protection and push directly when the merge is clean (or only version conflicts). Only creates a PR when there are real conflicts in other files. * chore: use app-slug instead of app-id for token generation * fix: address review feedback on sync-preview workflow - Pass PREVIEW_VERSION via env var instead of string interpolation in node -e scripts (safer against special chars) - Make git add of package-lock.json conditional on file existence to match the earlier -f guard - Replace loose title search for dedup with headRefName prefix filter to avoid false positives from unrelated PRs - Clarify why package.json/package-lock.json are special-cased (preview carries a different version string that needs preserving) * fix: restore preview-owned files after sync merge Adds a step to restore schemas/agentcore.schema.v1.json and CHANGELOG.md to preview's versions after merging main. These files are auto-generated during preview releases — schema-check CI rejects direct modifications to schemas/, and CHANGELOG.md tracks preview releases separately. * fix: use app-id instead of app-slug for GitHub App token Aligns with the pattern in PR #1210 and ci-failure-issue.yml. * feat: instrument telemetry for deploy command (CLI + TUI) (#1206) * chore: sync safe commits from main into preview Cherry-picks non-breaking changes from main: - docs: add telemetry instrumentation guide (#1197) - chore: replace PAT tokens with GitHub App token (#1198) - fix: add batch eval, recommendation, CloudWatch Logs permissions (#1113) - feat(evaluator): add kmsKeyArn support (#994) - chore: replace github.token with GitHub App token (#1210) - fix: sync-preview workflow rewrite (#1078) Skipped (requires dedicated effort due to Result type refactor): - refactor: unify result types with Result<T, E> (#1125) - feat: instrument telemetry for create/deploy (#1202, #1206) - feat: record command attrs on failure (#1204) * chore: merge main into preview with Result refactor Brings in all remaining main commits including: - refactor: unify result types with Result<T, E> (#1125) - feat: instrument telemetry for deploy command (#1206) - feat: record command attrs on failure (#1204) - feat: instrument telemetry for create command (#1202) Adapts preview's deploy flow to use OperationResult with string errors for compatibility with the telemetry layer. --------- Co-authored-by: Hweinstock <42325418+Hweinstock@users.noreply.github.com> Co-authored-by: Aidan Daly <99039782+aidandaly24@users.noreply.github.com> Co-authored-by: Gitika <53349492+notgitika@users.noreply.github.com> Co-authored-by: Aidan Daly <aidandal@amazon.com> Co-authored-by: Harrison Weinstock <hkobew@amazon.com> Co-authored-by: aws-aditya21 <saivenki@amazon.com> Co-authored-by: notgitika <gitijh@gmail.com>

notgitika · 2026-05-04T21:33:27Z

Summary

Adds missing IAM permissions to docs policy files for batch evaluation and recommendations:

Add BatchEvalAndRecommendations statement to iam-policy-user.json (9 actions)
Add CloudWatch Logs write permissions to LogsStreamingAndSearch (DescribeLogGroups, CreateLogGroup, CreateLogStream, PutLogEvents, PutRetentionPolicy)
Update PERMISSIONS.md with batch eval/recommendations reference section and logging entries

Also applied logs:PutRetentionPolicy to the e2e-github-actions role in the test account.

Context

E2E batch eval tests were failing with 403 on PutRetentionPolicy -- the batch eval API sets a retention policy on its results log group.

Test plan

iam-policy-user.json is valid JSON
prettier passes
Applied PutRetentionPolicy to e2e-github-actions role
Re-run E2E to confirm batch eval tests pass

…sions to docs

agentcore-cli-automation · 2026-05-04T21:36:53Z

Thanks for adding these permissions. One issue worth addressing before merge:

The CLI Commands column in PERMISSIONS.md references commands that don't exist in the CLI today.

In the new "Batch evaluation and recommendations" table:

run batch-eval — no such command. run only has the eval subcommand (see src/cli/commands/run/command.tsx).
run recommend — no such command.
evals history is listed for bedrock-agentcore:ListBatchEvaluations, but evals history is a local-only command that reads from disk via handleListEvalRuns → listEvalRuns() (see src/cli/operations/eval/list-eval-runs.ts and src/cli/commands/eval/command.tsx). It makes no AWS API call, so it doesn't require ListBatchEvaluations.

I also couldn't find any references to StartBatchEvaluation, StartRecommendation, etc. anywhere in src/ or the e2e tests — the run eval code path uses the Evaluate API (already covered).

If the service is invoking batch eval / recommendation APIs server-side on behalf of the user when run eval is called (which would explain the PutRetentionPolicy 403 you mentioned in the description), it might be worth:

Options to address this:

Update the CLI Commands column entries to point to the actual command that triggers these permissions (e.g. run evals if that's the command that indirectly causes batch eval invocations server-side), and update the Purpose column to explain it's service-invoked on behalf of the caller.
Drop the Batch evaluation and recommendations section from PERMISSIONS.md and the BatchEvalAndRecommendations statement from iam-policy-user.json for now if no CLI command actually calls these APIs — and only keep the logs permissions that are actually failing (PutRetentionPolicy, etc.).
If these commands are planned/upcoming, either add them in this PR or gate the doc changes until the commands land, so the docs don't advertise commands users can't run.

The logs:* additions (PutRetentionPolicy, CreateLogGroup, CreateLogStream, PutLogEvents, DescribeLogGroups) look fine as long as the batch-eval-via-service-principal explanation holds, though the CLI Commands column still lists the non-existent run batch-eval for most of them and should be fixed the same way.

github-actions · 2026-05-04T21:37:13Z

Coverage Report

Status	Category	Percentage	Covered / Total
🔵	Lines	43.04%	8990 / 20883
🔵	Statements	42.33%	9546 / 22550
🔵	Functions	39.91%	1552 / 3888
🔵	Branches	39.92%	5794 / 14511

Generated in workflow #2378 for commit 1b9ef28 by the Vitest Coverage Report Action

Cherry-picks non-breaking changes from main: - docs: add telemetry instrumentation guide (#1197) - chore: replace PAT tokens with GitHub App token (#1198) - fix: add batch eval, recommendation, CloudWatch Logs permissions (#1113) - feat(evaluator): add kmsKeyArn support (#994) - chore: replace github.token with GitHub App token (#1210) - fix: sync-preview workflow rewrite (#1078) Skipped (requires dedicated effort due to Result type refactor): - refactor: unify result types with Result<T, E> (#1125) - feat: instrument telemetry for create/deploy (#1202, #1206) - feat: record command attrs on failure (#1204)

* docs: add telemetry instrumentation guide (#1197) * chore: replace PAT tokens with GitHub App token (#1198) Replace secrets.PAT_TOKEN and secrets.AUTOMATION_ACCOUNT_PAT_TOKEN with short-lived tokens generated by the agentcore-devx-automation GitHub App (ID: 3637953) via actions/create-github-app-token@v1. This improves security by using ephemeral tokens scoped to the installation rather than long-lived personal access tokens. Requires adding repo variable APP_ID=3637953 and repo secret APP_PRIVATE_KEY with the app's RSA private key. * fix: add batch eval, recommendation, and CloudWatch Logs write permissions to docs (#1113) * feat: instrument telemetry for create command (CLI + TUI) (#1202) Add telemetry recording to the create command in both CLI and TUI paths: - CLI: wrap handleCreateCLI with runCliCommand to emit CreateAttrs on success/failure - TUI: wrap useCreateFlow's run() with withCommandRunTelemetry - Add telemetry assertions to existing integration tests (frameworks + edge cases) * chore: replace all github.token/GITHUB_TOKEN with GitHub App token Replaces all occurrences of \${{ github.token }} and \${{ secrets.GITHUB_TOKEN }} across .github/workflows/ with a per-job GitHub App token generated via actions/create-github-app-token@v1 using vars.APP_ID and secrets.APP_PRIVATE_KEY. * fix: bump versions to resolve security audit failure * revert: keep pr-title.yml using GITHUB_TOKEN (read-only access sufficient) * feat(evaluator): add kmsKeyArn support for custom evaluator (#994) * feat(evaluator): Add kmsKeyArn support for custom evaluator * fix: sync package-lock.json with package.json The lock file was out of sync after dependency bumps on main were merged, causing npm ci to fail in CI. * fix: revert unrelated dep bumps and fix formatting Reverts @opentelemetry/exporter-metrics-otlp-http ^0.217.0 back to ^0.214.0 and secretlint ^13.0.0 back to ^12.2.0 — these were accidentally included in the feature commit from unmerged dependabot PRs and introduce high-severity protobufjs vulnerabilities. Restores fast-xml-parser and @aws-sdk/xml-builder overrides that were also inadvertently removed. Fixes Prettier formatting on agentcore-project.ts import lines. * fix: sync package-lock.json with updated dependencies --------- Co-authored-by: notgitika <gitijh@gmail.com> * refactor: unify result types with discriminated Result<T, E> union (#1125) * refactor: unify result types with discriminated Result<T, E> union Introduce a shared Result<T, E> type (inspired by Rust's Result) that replaces ad-hoc { success: boolean; error?: string } patterns across the codebase. Key changes: - Add src/lib/types.ts with Result<T, E> discriminated union type - Add toError() helper in src/cli/errors.ts for catch blocks - Migrate all command, operation, and primitive result types to Result<T> - Error field is now Error (not string) on the failure branch - Data fields only exist on the success branch (proper narrowing) - Update all consumers to narrow before accessing branch-specific fields - Update test assertions to match new Error objects and add narrowing * docs: update AGENTS.md and telemetry README to reflect Result<T, E> type * feat: record command attrs on telemetry failure via fallbackAttrs (#1204) * feat: record command attrs on telemetry failure via fallbackAttrs Add optional fallbackAttrs parameter to client.withCommandRun so command-specific attributes are recorded even when the callback throws. - client.ts: accept fallbackAttrs, use on failure instead of {} - client.ts: run resilientParse on all non-empty attrs (not just success) - cli-command-run.ts: withCommandRunTelemetry passes attrs as fallbackAttrs - cli-command-run.ts: runCliCommand accepts optional knownAttrs param - command.tsx: extract knownAttrs upfront, pass to runCliCommand - client.test.ts: add unit tests for fallbackAttrs behavior - create-edge-cases.test.ts: assert attrs present on failure entry * chore: rebase onto mainline * fix: sync-preview pushes directly on clean merge, PRs only on conflict (#1078) * fix: sync-preview workflow restores version instead of ignoring files Instead of keeping preview's entire package.json/package-lock.json (which discards new deps, scripts, etc. from main), accept main's content and surgically restore only the version field to preview's value after merge. * fix: push directly to preview on clean merge via GitHub App bypass Use agentcore-devx-automation app token to bypass branch protection and push directly when the merge is clean (or only version conflicts). Only creates a PR when there are real conflicts in other files. * chore: use app-slug instead of app-id for token generation * fix: address review feedback on sync-preview workflow - Pass PREVIEW_VERSION via env var instead of string interpolation in node -e scripts (safer against special chars) - Make git add of package-lock.json conditional on file existence to match the earlier -f guard - Replace loose title search for dedup with headRefName prefix filter to avoid false positives from unrelated PRs - Clarify why package.json/package-lock.json are special-cased (preview carries a different version string that needs preserving) * fix: restore preview-owned files after sync merge Adds a step to restore schemas/agentcore.schema.v1.json and CHANGELOG.md to preview's versions after merging main. These files are auto-generated during preview releases — schema-check CI rejects direct modifications to schemas/, and CHANGELOG.md tracks preview releases separately. * fix: use app-id instead of app-slug for GitHub App token Aligns with the pattern in PR #1210 and ci-failure-issue.yml. * feat: instrument telemetry for deploy command (CLI + TUI) (#1206) * chore: sync safe commits from main into preview Cherry-picks non-breaking changes from main: - docs: add telemetry instrumentation guide (#1197) - chore: replace PAT tokens with GitHub App token (#1198) - fix: add batch eval, recommendation, CloudWatch Logs permissions (#1113) - feat(evaluator): add kmsKeyArn support (#994) - chore: replace github.token with GitHub App token (#1210) - fix: sync-preview workflow rewrite (#1078) Skipped (requires dedicated effort due to Result type refactor): - refactor: unify result types with Result<T, E> (#1125) - feat: instrument telemetry for create/deploy (#1202, #1206) - feat: record command attrs on failure (#1204) * chore: merge main into preview with Result refactor Brings in all remaining main commits including: - refactor: unify result types with Result<T, E> (#1125) - feat: instrument telemetry for deploy command (#1206) - feat: record command attrs on failure (#1204) - feat: instrument telemetry for create command (#1202) Adapts preview's deploy flow to use OperationResult with string errors for compatibility with the telemetry layer. --------- Co-authored-by: Hweinstock <42325418+Hweinstock@users.noreply.github.com> Co-authored-by: Aidan Daly <99039782+aidandaly24@users.noreply.github.com> Co-authored-by: Gitika <53349492+notgitika@users.noreply.github.com> Co-authored-by: Aidan Daly <aidandal@amazon.com> Co-authored-by: Harrison Weinstock <hkobew@amazon.com> Co-authored-by: aws-aditya21 <saivenki@amazon.com> Co-authored-by: notgitika <gitijh@gmail.com>

fix: add batch eval, recommendation, and CloudWatch Logs write permis…
1b9ef28
…sions to docs

notgitika requested a review from a teamMay 4, 2026 21:33

github-actionsBot added size/s PR size: S agentcore-harness-reviewing AgentCore Harness review in progress labels May 4, 2026

github-actionsBot removed the agentcore-harness-reviewing AgentCore Harness review in progress label May 4, 2026

aidandaly24 approved these changes May 11, 2026
View reviewed changes

notgitika merged commit ce50d52 into mainMay 11, 2026
26 checks passed

notgitika deleted the fix/add-put-retention-policy-permission branch May 11, 2026 20:08

fix: add batch eval, recommendation, and CloudWatch Logs write permis…
1b9ef28
…sions to docs

notgitika requested a review from a teamMay 4, 2026 21:33

github-actionsBot added size/s PR size: S agentcore-harness-reviewing AgentCore Harness review in progress labels May 4, 2026

github-actionsBot removed the agentcore-harness-reviewing AgentCore Harness review in progress label May 4, 2026

aidandaly24 approved these changes May 11, 2026
View reviewed changes

notgitika merged commit ce50d52 into mainMay 11, 2026
26 checks passed

notgitika deleted the fix/add-put-retention-policy-permission branch May 11, 2026 20:08

fix: add batch eval, recommendation, and CloudWatch Logs write permis…
1b9ef28
…sions to docs

notgitika requested a review from a teamMay 4, 2026 21:33

github-actionsBot added size/s PR size: S agentcore-harness-reviewing AgentCore Harness review in progress labels May 4, 2026

github-actionsBot removed the agentcore-harness-reviewing AgentCore Harness review in progress label May 4, 2026

aidandaly24 approved these changes May 11, 2026
View reviewed changes

notgitika merged commit ce50d52 into mainMay 11, 2026
26 checks passed

notgitika deleted the fix/add-put-retention-policy-permission branch May 11, 2026 20:08

fix: add batch eval, recommendation, and CloudWatch Logs write permis…
1b9ef28
…sions to docs

notgitika requested a review from a teamMay 4, 2026 21:33

github-actionsBot added size/s PR size: S agentcore-harness-reviewing AgentCore Harness review in progress labels May 4, 2026

github-actionsBot removed the agentcore-harness-reviewing AgentCore Harness review in progress label May 4, 2026

aidandaly24 approved these changes May 11, 2026
View reviewed changes

notgitika merged commit ce50d52 into mainMay 11, 2026
26 checks passed

notgitika deleted the fix/add-put-retention-policy-permission branch May 11, 2026 20:08

fix: add batch eval, recommendation, and CloudWatch Logs write permis…
1b9ef28
…sions to docs

notgitika requested a review from a teamMay 4, 2026 21:33

github-actionsBot added size/s PR size: S agentcore-harness-reviewing AgentCore Harness review in progress labels May 4, 2026

github-actionsBot removed the agentcore-harness-reviewing AgentCore Harness review in progress label May 4, 2026

aidandaly24 approved these changes May 11, 2026
View reviewed changes

notgitika merged commit ce50d52 into mainMay 11, 2026
26 checks passed

notgitika deleted the fix/add-put-retention-policy-permission branch May 11, 2026 20:08

fix: add batch eval, recommendation, and CloudWatch Logs write permis…
1b9ef28
…sions to docs

notgitika requested a review from a teamMay 4, 2026 21:33

github-actionsBot added size/s PR size: S agentcore-harness-reviewing AgentCore Harness review in progress labels May 4, 2026

github-actionsBot removed the agentcore-harness-reviewing AgentCore Harness review in progress label May 4, 2026

aidandaly24 approved these changes May 11, 2026
View reviewed changes

notgitika merged commit ce50d52 into mainMay 11, 2026
26 checks passed

notgitika deleted the fix/add-put-retention-policy-permission branch May 11, 2026 20:08

fix: add batch eval, recommendation, and CloudWatch Logs write permis…
1b9ef28
…sions to docs

notgitika requested a review from a teamMay 4, 2026 21:33

github-actionsBot added size/s PR size: S agentcore-harness-reviewing AgentCore Harness review in progress labels May 4, 2026

github-actionsBot removed the agentcore-harness-reviewing AgentCore Harness review in progress label May 4, 2026

aidandaly24 approved these changes May 11, 2026
View reviewed changes

notgitika merged commit ce50d52 into mainMay 11, 2026
26 checks passed

notgitika deleted the fix/add-put-retention-policy-permission branch May 11, 2026 20:08

fix: add batch eval, recommendation, and CloudWatch Logs write permis…
1b9ef28
…sions to docs

notgitika requested a review from a teamMay 4, 2026 21:33

github-actionsBot added size/s PR size: S agentcore-harness-reviewing AgentCore Harness review in progress labels May 4, 2026

github-actionsBot removed the agentcore-harness-reviewing AgentCore Harness review in progress label May 4, 2026

aidandaly24 approved these changes May 11, 2026
View reviewed changes

notgitika merged commit ce50d52 into mainMay 11, 2026
26 checks passed

notgitika deleted the fix/add-put-retention-policy-permission branch May 11, 2026 20:08

Uh oh!

Conversation

notgitika commented May 4, 2026

Summary

Context

Test plan

Uh oh!

agentcore-cli-automation commented May 4, 2026

Uh oh!

github-actionsBot commented May 4, 2026

Coverage Report

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

notgitika commented May 4, 2026

Summary

Context

Test plan

Uh oh!

agentcore-cli-automation commented May 4, 2026

Uh oh!

github-actionsBot commented May 4, 2026

Coverage Report

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

notgitika commented May 4, 2026

Summary

Context

Test plan

Uh oh!

agentcore-cli-automation commented May 4, 2026

Uh oh!

github-actionsBot commented May 4, 2026

Coverage Report

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

notgitika commented May 4, 2026

Summary

Context

Test plan

Uh oh!

agentcore-cli-automation commented May 4, 2026

Uh oh!

github-actionsBot commented May 4, 2026

Coverage Report

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects