ci: cut full e2e time in half via vitest sharding by Hweinstock · Pull Request #1016 · aws/agentcore-cli

Hweinstock · 2026-04-29T14:40:51Z

Description

The full E2E suite takes 19-25 min because all 17 test files run on a single runner per CDK source. Each test deploys its own CloudFormation stack, and measured timings show parallel deploys have zero degradation (77s stays 77s with concurrent stacks).

This adds 6-way vitest sharding via the GitHub Actions matrix, going from 2 to 12 parallel runners. Also isolates import test resources per run to prevent conflicts when npm and main matrix entries hit the same AWS account concurrently.

Measured results:

Shards	Wall-clock	Slowest shard
1 (baseline)	19-25 min	—
4	~15 min	890s
6	~11.5 min	691s
8	~12.0 min	717s

6 shards appears to be the sweet spot. Adding more shards creates diminishing returns since the main bottleneck is the deploy (274s) and container build (204s). This is roughly what the e2e tests take on PRs when running minimal files so we're likely close to how fast we can get it.

Inspired by #989.

Docs: https://vitest.dev/guide/improving-performance.html#sharding

Import test resource isolation

The import-resources.test.ts setup creates AWS resources (runtime, memory, evaluator) via Python scripts, then saves their ARNs to bugbash-resources.json. The cleanup script deletes those resources and also calls cleanup_s3_code_objects() which previously deleted all objects from the shared S3 bucket bugbash-agentcore-code-{account}-{region}.

The npm and main matrix entries are separate runners hitting the same AWS account simultaneously. Both run import-resources.test.ts. When one job's cleanup finishes first, it nuked all S3 objects — including the code.zip the other job uploaded and was still using, causing downstream resource creation failures.

The fix: each run now gets a unique RESOURCE_SUFFIX (the test's timestamp-based suffix). This scopes:

The resources file: bugbash-resources-{suffix}.json instead of bugbash-resources.json
The S3 prefix: bugbash-{suffix}/code.zip instead of bugbash/code.zip
The S3 cleanup: only deletes objects under its own prefix

Related Issue

N/A — CI performance improvement.

Documentation PR

N/A

Type of Change

Testing

Tried different shard sizes, runs are here: https://github.com/aws/agentcore-cli/actions/runs/25165632571, https://github.com/aws/agentcore-cli/actions/runs/25166733599

I ran npm run test:unit and npm run test:integ
I ran npm run typecheck
I ran npm run lint
If I modified src/assets/, I ran npm run test:update-snapshots and committed the updated snapshots

Checklist

I have read the CONTRIBUTING document
I have added any necessary tests that prove my fix is effective or my feature works
I have updated the documentation accordingly
I have added an appropriate example to the documentation to outline the feature, or no new docs are needed
My changes generate no new warnings
Any dependent changes have been merged and published

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the
terms of your choice.

agentcore-cli-automation · 2026-04-29T14:42:54Z

Reviewed — looks good to merge.

Verified:

vitest run --project e2e --shard=X/4 is a supported combination in vitest 4.x
16 e2e test files distribute across 4 shards (~4 files each); imbalance is acceptable given fail-fast: false and 60-minute timeout
Parallel test concerns look fine: cleanupStaleCredentialProviders() uses a 30-min cutoff so it won't touch resources from concurrently running shards; agent names include framework/provider prefixes so they won't collide across files
No production code touched; failure mode is just a CI run that surfaces the issue, easy to revert

One thing worth noting (not blocking): your local measurement validated 2 concurrent deploys, but this moves to 8 concurrent (4 shards × 2 cdk-sources). If you see intermittent CloudFormation/AgentCore control-plane throttling in the first few scheduled runs, that'd be the likely cause — but worth finding out empirically rather than preemptively capping.

- Add 6-way vitest sharding to the cdk-source matrix (2 → 12 parallel runners) - Isolate import test resources per run via RESOURCE_SUFFIX to prevent concurrent conflicts

Hweinstock · 2026-04-30T13:17:17Z

/strands review

github-actions · 2026-04-30T13:49:31Z

Code Review Summary

I've reviewed PR #1016, which adds 6-way vitest sharding to reduce E2E test time from 19-25 min to ~11.5 min. The approach is sound and well-measured, but there is one blocking issue that must be fixed before merge.

🔴 Blocking Issues

Missing import in `cleanup_resources.py`

File:e2e-tests/fixtures/import/cleanup_resources.py
Lines: 15, 25

The script uses RESOURCE_SUFFIX on line 25 but doesn't import it from common.py on line 15:

# Line 15 - current importsfromcommonimportREGION, RESOURCES_FILE, get_control_client, get_account_id# Line 25 - uses RESOURCE_SUFFIX (not imported!)prefix=f"bugbash-{RESOURCE_SUFFIX}/"ifRESOURCE_SUFFIXelse""

Impact: This will cause a NameError: name 'RESOURCE_SUFFIX' is not defined when cleanup runs after a test, potentially leaving resources in AWS.

Fix: Add RESOURCE_SUFFIX to the import statement:

fromcommonimportREGION, RESOURCES_FILE, RESOURCE_SUFFIX, get_control_client, get_account_id

💡 Suggestions (non-blocking)

1. Resource suffix collision risk

Files:e2e-tests/import-resources.test.ts:37, e2e-tests/import-gateway.test.ts:34

Both tests generate the suffix using Date.now():

constsuffix=Date.now().toString().slice(-8);

In a sharded run, if two shards start within the same millisecond and run the same test, they could collide on resource names. While unlikely, using a more unique identifier would be more robust:

constsuffix=`${Date.now()}-${randomUUID().slice(0,8)}`;

Or include the shard number if available in the environment.

2. S3 cleanup safety concern

File:e2e-tests/fixtures/import/cleanup_resources.py:25

When RESOURCE_SUFFIX is empty or unset, the prefix becomes "", which means list_objects_v2 will return all objects in the bucket:

prefix=f"bugbash-{RESOURCE_SUFFIX}/"ifRESOURCE_SUFFIXelse""list_args= {"Bucket": bucket_name}
ifprefix: # Empty string is falsy, so this branch is skippedlist_args["Prefix"] =prefixresp=s3.list_objects_v2(**list_args) # Lists ALL objects

This could accidentally delete objects from other concurrent runs if the env var is missing. Consider:

Making RESOURCE_SUFFIX required (fail fast if unset)
Or use a fallback like "default" instead of empty string

3. Documentation: Why 6 shards?

File:.github/workflows/e2e-tests-full.yml:30

The PR description explains the measured results for 4, 6, and 8 shards, and that 6 is optimal. Consider adding a brief comment in the workflow file explaining why 6 was chosen:

matrix:
cdk-source: [npm, main]# 6 shards provides optimal balance (~11.5 min wall-clock)# 8 shards showed no improvement due to memory deploy (274s) and container build (204s) bottlenecksshard: ['1/6', '2/6', '3/6', '4/6', '5/6', '6/6']

❓ Questions

Concurrency increase: 2 → 12 parallel deploys

File:.github/workflows/e2e-tests-full.yml:28-30

The PR moves from 2 concurrent runners (2 cdk-source entries) to 12 (2 × 6 shards). The description mentions testing 2 concurrent deploys locally but not 12.

Question: Have you verified that the AWS account's CloudFormation/AgentCore control plane can handle 12 concurrent stacks without throttling? The bot comment mentioned this could be "worth finding out empirically," but it would be good to confirm if you've already done a test run with the full matrix.

✅ Positive Aspects

Well-measured performance improvement: Clear data showing 19-25 min → 11.5 min
Proper resource isolation: Using RESOURCE_SUFFIX to scope S3 objects and resources files per run prevents conflicts between npm and main matrix entries
No production code changes: Only CI and test infrastructure, easy to revert if issues arise
Comprehensive testing: The PR description shows you tested 4, 6, and 8 shards to find the optimal configuration

Next Steps

Fix the blocking import issue in cleanup_resources.py
Consider the suggestions above (especially the S3 cleanup safety concern)
Confirm the 12 concurrent deploys won't throttle in CI

Overall, this is a great performance improvement with a solid approach. Once the import bug is fixed, this should be good to merge! 🚀

Hweinstock · 2026-04-30T16:03:49Z

Addressed comments and re ran https://github.com/aws/agentcore-cli/actions/runs/25175265881.

Hweinstock · 2026-04-29T14:40:51Z

Description

The full E2E suite takes 19-25 min because all 17 test files run on a single runner per CDK source. Each test deploys its own CloudFormation stack, and measured timings show parallel deploys have zero degradation (77s stays 77s with concurrent stacks).

This adds 6-way vitest sharding via the GitHub Actions matrix, going from 2 to 12 parallel runners. Also isolates import test resources per run to prevent conflicts when npm and main matrix entries hit the same AWS account concurrently.

Measured results:

Shards	Wall-clock	Slowest shard
1 (baseline)	19-25 min	—
4	~15 min	890s
6	~11.5 min	691s
8	~12.0 min	717s

6 shards appears to be the sweet spot. Adding more shards creates diminishing returns since the main bottleneck is the deploy (274s) and container build (204s). This is roughly what the e2e tests take on PRs when running minimal files so we're likely close to how fast we can get it.

Inspired by #989.

Docs: https://vitest.dev/guide/improving-performance.html#sharding

Import test resource isolation

The import-resources.test.ts setup creates AWS resources (runtime, memory, evaluator) via Python scripts, then saves their ARNs to bugbash-resources.json. The cleanup script deletes those resources and also calls cleanup_s3_code_objects() which previously deleted all objects from the shared S3 bucket bugbash-agentcore-code-{account}-{region}.

The npm and main matrix entries are separate runners hitting the same AWS account simultaneously. Both run import-resources.test.ts. When one job's cleanup finishes first, it nuked all S3 objects — including the code.zip the other job uploaded and was still using, causing downstream resource creation failures.

The fix: each run now gets a unique RESOURCE_SUFFIX (the test's timestamp-based suffix). This scopes:

The resources file: bugbash-resources-{suffix}.json instead of bugbash-resources.json
The S3 prefix: bugbash-{suffix}/code.zip instead of bugbash/code.zip
The S3 cleanup: only deletes objects under its own prefix

Related Issue

N/A — CI performance improvement.

Documentation PR

N/A

Type of Change

Testing

Tried different shard sizes, runs are here: https://github.com/aws/agentcore-cli/actions/runs/25165632571, https://github.com/aws/agentcore-cli/actions/runs/25166733599

I ran npm run test:unit and npm run test:integ
I ran npm run typecheck
I ran npm run lint
If I modified src/assets/, I ran npm run test:update-snapshots and committed the updated snapshots

Checklist

I have read the CONTRIBUTING document
I have added any necessary tests that prove my fix is effective or my feature works
I have updated the documentation accordingly
I have added an appropriate example to the documentation to outline the feature, or no new docs are needed
My changes generate no new warnings
Any dependent changes have been merged and published

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the
terms of your choice.

agentcore-cli-automation · 2026-04-29T14:42:54Z

Reviewed — looks good to merge.

Verified:

vitest run --project e2e --shard=X/4 is a supported combination in vitest 4.x
16 e2e test files distribute across 4 shards (~4 files each); imbalance is acceptable given fail-fast: false and 60-minute timeout
Parallel test concerns look fine: cleanupStaleCredentialProviders() uses a 30-min cutoff so it won't touch resources from concurrently running shards; agent names include framework/provider prefixes so they won't collide across files
No production code touched; failure mode is just a CI run that surfaces the issue, easy to revert

One thing worth noting (not blocking): your local measurement validated 2 concurrent deploys, but this moves to 8 concurrent (4 shards × 2 cdk-sources). If you see intermittent CloudFormation/AgentCore control-plane throttling in the first few scheduled runs, that'd be the likely cause — but worth finding out empirically rather than preemptively capping.

- Add 6-way vitest sharding to the cdk-source matrix (2 → 12 parallel runners) - Isolate import test resources per run via RESOURCE_SUFFIX to prevent concurrent conflicts

Hweinstock · 2026-04-30T13:17:17Z

/strands review

github-actions · 2026-04-30T13:49:31Z

Code Review Summary

I've reviewed PR #1016, which adds 6-way vitest sharding to reduce E2E test time from 19-25 min to ~11.5 min. The approach is sound and well-measured, but there is one blocking issue that must be fixed before merge.

🔴 Blocking Issues

Missing import in `cleanup_resources.py`

File:e2e-tests/fixtures/import/cleanup_resources.py
Lines: 15, 25

The script uses RESOURCE_SUFFIX on line 25 but doesn't import it from common.py on line 15:

# Line 15 - current importsfromcommonimportREGION, RESOURCES_FILE, get_control_client, get_account_id# Line 25 - uses RESOURCE_SUFFIX (not imported!)prefix=f"bugbash-{RESOURCE_SUFFIX}/"ifRESOURCE_SUFFIXelse""

Impact: This will cause a NameError: name 'RESOURCE_SUFFIX' is not defined when cleanup runs after a test, potentially leaving resources in AWS.

Fix: Add RESOURCE_SUFFIX to the import statement:

fromcommonimportREGION, RESOURCES_FILE, RESOURCE_SUFFIX, get_control_client, get_account_id

💡 Suggestions (non-blocking)

1. Resource suffix collision risk

Files:e2e-tests/import-resources.test.ts:37, e2e-tests/import-gateway.test.ts:34

Both tests generate the suffix using Date.now():

constsuffix=Date.now().toString().slice(-8);

In a sharded run, if two shards start within the same millisecond and run the same test, they could collide on resource names. While unlikely, using a more unique identifier would be more robust:

constsuffix=`${Date.now()}-${randomUUID().slice(0,8)}`;

Or include the shard number if available in the environment.

2. S3 cleanup safety concern

File:e2e-tests/fixtures/import/cleanup_resources.py:25

When RESOURCE_SUFFIX is empty or unset, the prefix becomes "", which means list_objects_v2 will return all objects in the bucket:

prefix=f"bugbash-{RESOURCE_SUFFIX}/"ifRESOURCE_SUFFIXelse""list_args= {"Bucket": bucket_name}
ifprefix: # Empty string is falsy, so this branch is skippedlist_args["Prefix"] =prefixresp=s3.list_objects_v2(**list_args) # Lists ALL objects

This could accidentally delete objects from other concurrent runs if the env var is missing. Consider:

Making RESOURCE_SUFFIX required (fail fast if unset)
Or use a fallback like "default" instead of empty string

3. Documentation: Why 6 shards?

File:.github/workflows/e2e-tests-full.yml:30

The PR description explains the measured results for 4, 6, and 8 shards, and that 6 is optimal. Consider adding a brief comment in the workflow file explaining why 6 was chosen:

matrix:
cdk-source: [npm, main]# 6 shards provides optimal balance (~11.5 min wall-clock)# 8 shards showed no improvement due to memory deploy (274s) and container build (204s) bottlenecksshard: ['1/6', '2/6', '3/6', '4/6', '5/6', '6/6']

❓ Questions

Concurrency increase: 2 → 12 parallel deploys

File:.github/workflows/e2e-tests-full.yml:28-30

The PR moves from 2 concurrent runners (2 cdk-source entries) to 12 (2 × 6 shards). The description mentions testing 2 concurrent deploys locally but not 12.

Question: Have you verified that the AWS account's CloudFormation/AgentCore control plane can handle 12 concurrent stacks without throttling? The bot comment mentioned this could be "worth finding out empirically," but it would be good to confirm if you've already done a test run with the full matrix.

✅ Positive Aspects

Well-measured performance improvement: Clear data showing 19-25 min → 11.5 min
Proper resource isolation: Using RESOURCE_SUFFIX to scope S3 objects and resources files per run prevents conflicts between npm and main matrix entries
No production code changes: Only CI and test infrastructure, easy to revert if issues arise
Comprehensive testing: The PR description shows you tested 4, 6, and 8 shards to find the optimal configuration

Next Steps

Fix the blocking import issue in cleanup_resources.py
Consider the suggestions above (especially the S3 cleanup safety concern)
Confirm the 12 concurrent deploys won't throttle in CI

Overall, this is a great performance improvement with a solid approach. Once the import bug is fixed, this should be good to merge! 🚀

Hweinstock · 2026-04-30T16:03:49Z

Addressed comments and re ran https://github.com/aws/agentcore-cli/actions/runs/25175265881.

Hweinstock · 2026-04-29T14:40:51Z

Description

The full E2E suite takes 19-25 min because all 17 test files run on a single runner per CDK source. Each test deploys its own CloudFormation stack, and measured timings show parallel deploys have zero degradation (77s stays 77s with concurrent stacks).

This adds 6-way vitest sharding via the GitHub Actions matrix, going from 2 to 12 parallel runners. Also isolates import test resources per run to prevent conflicts when npm and main matrix entries hit the same AWS account concurrently.

Measured results:

Shards	Wall-clock	Slowest shard
1 (baseline)	19-25 min	—
4	~15 min	890s
6	~11.5 min	691s
8	~12.0 min	717s

6 shards appears to be the sweet spot. Adding more shards creates diminishing returns since the main bottleneck is the deploy (274s) and container build (204s). This is roughly what the e2e tests take on PRs when running minimal files so we're likely close to how fast we can get it.

Inspired by #989.

Docs: https://vitest.dev/guide/improving-performance.html#sharding

Import test resource isolation

The import-resources.test.ts setup creates AWS resources (runtime, memory, evaluator) via Python scripts, then saves their ARNs to bugbash-resources.json. The cleanup script deletes those resources and also calls cleanup_s3_code_objects() which previously deleted all objects from the shared S3 bucket bugbash-agentcore-code-{account}-{region}.

The npm and main matrix entries are separate runners hitting the same AWS account simultaneously. Both run import-resources.test.ts. When one job's cleanup finishes first, it nuked all S3 objects — including the code.zip the other job uploaded and was still using, causing downstream resource creation failures.

The fix: each run now gets a unique RESOURCE_SUFFIX (the test's timestamp-based suffix). This scopes:

The resources file: bugbash-resources-{suffix}.json instead of bugbash-resources.json
The S3 prefix: bugbash-{suffix}/code.zip instead of bugbash/code.zip
The S3 cleanup: only deletes objects under its own prefix

Related Issue

N/A — CI performance improvement.

Documentation PR

N/A

Type of Change

Testing

Tried different shard sizes, runs are here: https://github.com/aws/agentcore-cli/actions/runs/25165632571, https://github.com/aws/agentcore-cli/actions/runs/25166733599

I ran npm run test:unit and npm run test:integ
I ran npm run typecheck
I ran npm run lint
If I modified src/assets/, I ran npm run test:update-snapshots and committed the updated snapshots

Checklist

I have read the CONTRIBUTING document
I have added any necessary tests that prove my fix is effective or my feature works
I have updated the documentation accordingly
I have added an appropriate example to the documentation to outline the feature, or no new docs are needed
My changes generate no new warnings
Any dependent changes have been merged and published

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the
terms of your choice.

agentcore-cli-automation · 2026-04-29T14:42:54Z

Reviewed — looks good to merge.

Verified:

vitest run --project e2e --shard=X/4 is a supported combination in vitest 4.x
16 e2e test files distribute across 4 shards (~4 files each); imbalance is acceptable given fail-fast: false and 60-minute timeout
Parallel test concerns look fine: cleanupStaleCredentialProviders() uses a 30-min cutoff so it won't touch resources from concurrently running shards; agent names include framework/provider prefixes so they won't collide across files
No production code touched; failure mode is just a CI run that surfaces the issue, easy to revert

One thing worth noting (not blocking): your local measurement validated 2 concurrent deploys, but this moves to 8 concurrent (4 shards × 2 cdk-sources). If you see intermittent CloudFormation/AgentCore control-plane throttling in the first few scheduled runs, that'd be the likely cause — but worth finding out empirically rather than preemptively capping.

- Add 6-way vitest sharding to the cdk-source matrix (2 → 12 parallel runners) - Isolate import test resources per run via RESOURCE_SUFFIX to prevent concurrent conflicts

Hweinstock · 2026-04-30T13:17:17Z

/strands review

github-actions · 2026-04-30T13:49:31Z

Code Review Summary

I've reviewed PR #1016, which adds 6-way vitest sharding to reduce E2E test time from 19-25 min to ~11.5 min. The approach is sound and well-measured, but there is one blocking issue that must be fixed before merge.

🔴 Blocking Issues

Missing import in `cleanup_resources.py`

File:e2e-tests/fixtures/import/cleanup_resources.py
Lines: 15, 25

The script uses RESOURCE_SUFFIX on line 25 but doesn't import it from common.py on line 15:

# Line 15 - current importsfromcommonimportREGION, RESOURCES_FILE, get_control_client, get_account_id# Line 25 - uses RESOURCE_SUFFIX (not imported!)prefix=f"bugbash-{RESOURCE_SUFFIX}/"ifRESOURCE_SUFFIXelse""

Impact: This will cause a NameError: name 'RESOURCE_SUFFIX' is not defined when cleanup runs after a test, potentially leaving resources in AWS.

Fix: Add RESOURCE_SUFFIX to the import statement:

fromcommonimportREGION, RESOURCES_FILE, RESOURCE_SUFFIX, get_control_client, get_account_id

💡 Suggestions (non-blocking)

1. Resource suffix collision risk

Files:e2e-tests/import-resources.test.ts:37, e2e-tests/import-gateway.test.ts:34

Both tests generate the suffix using Date.now():

constsuffix=Date.now().toString().slice(-8);

In a sharded run, if two shards start within the same millisecond and run the same test, they could collide on resource names. While unlikely, using a more unique identifier would be more robust:

constsuffix=`${Date.now()}-${randomUUID().slice(0,8)}`;

Or include the shard number if available in the environment.

2. S3 cleanup safety concern

File:e2e-tests/fixtures/import/cleanup_resources.py:25

When RESOURCE_SUFFIX is empty or unset, the prefix becomes "", which means list_objects_v2 will return all objects in the bucket:

prefix=f"bugbash-{RESOURCE_SUFFIX}/"ifRESOURCE_SUFFIXelse""list_args= {"Bucket": bucket_name}
ifprefix: # Empty string is falsy, so this branch is skippedlist_args["Prefix"] =prefixresp=s3.list_objects_v2(**list_args) # Lists ALL objects

This could accidentally delete objects from other concurrent runs if the env var is missing. Consider:

Making RESOURCE_SUFFIX required (fail fast if unset)
Or use a fallback like "default" instead of empty string

3. Documentation: Why 6 shards?

File:.github/workflows/e2e-tests-full.yml:30

The PR description explains the measured results for 4, 6, and 8 shards, and that 6 is optimal. Consider adding a brief comment in the workflow file explaining why 6 was chosen:

matrix:
cdk-source: [npm, main]# 6 shards provides optimal balance (~11.5 min wall-clock)# 8 shards showed no improvement due to memory deploy (274s) and container build (204s) bottlenecksshard: ['1/6', '2/6', '3/6', '4/6', '5/6', '6/6']

❓ Questions

Concurrency increase: 2 → 12 parallel deploys

File:.github/workflows/e2e-tests-full.yml:28-30

The PR moves from 2 concurrent runners (2 cdk-source entries) to 12 (2 × 6 shards). The description mentions testing 2 concurrent deploys locally but not 12.

Question: Have you verified that the AWS account's CloudFormation/AgentCore control plane can handle 12 concurrent stacks without throttling? The bot comment mentioned this could be "worth finding out empirically," but it would be good to confirm if you've already done a test run with the full matrix.

✅ Positive Aspects

Well-measured performance improvement: Clear data showing 19-25 min → 11.5 min
Proper resource isolation: Using RESOURCE_SUFFIX to scope S3 objects and resources files per run prevents conflicts between npm and main matrix entries
No production code changes: Only CI and test infrastructure, easy to revert if issues arise
Comprehensive testing: The PR description shows you tested 4, 6, and 8 shards to find the optimal configuration

Next Steps

Fix the blocking import issue in cleanup_resources.py
Consider the suggestions above (especially the S3 cleanup safety concern)
Confirm the 12 concurrent deploys won't throttle in CI

Overall, this is a great performance improvement with a solid approach. Once the import bug is fixed, this should be good to merge! 🚀

Hweinstock · 2026-04-30T16:03:49Z

Addressed comments and re ran https://github.com/aws/agentcore-cli/actions/runs/25175265881.

Hweinstock · 2026-04-29T14:40:51Z

Description

The full E2E suite takes 19-25 min because all 17 test files run on a single runner per CDK source. Each test deploys its own CloudFormation stack, and measured timings show parallel deploys have zero degradation (77s stays 77s with concurrent stacks).

This adds 6-way vitest sharding via the GitHub Actions matrix, going from 2 to 12 parallel runners. Also isolates import test resources per run to prevent conflicts when npm and main matrix entries hit the same AWS account concurrently.

Measured results:

Shards	Wall-clock	Slowest shard
1 (baseline)	19-25 min	—
4	~15 min	890s
6	~11.5 min	691s
8	~12.0 min	717s

6 shards appears to be the sweet spot. Adding more shards creates diminishing returns since the main bottleneck is the deploy (274s) and container build (204s). This is roughly what the e2e tests take on PRs when running minimal files so we're likely close to how fast we can get it.

Inspired by #989.

Docs: https://vitest.dev/guide/improving-performance.html#sharding

Import test resource isolation

The import-resources.test.ts setup creates AWS resources (runtime, memory, evaluator) via Python scripts, then saves their ARNs to bugbash-resources.json. The cleanup script deletes those resources and also calls cleanup_s3_code_objects() which previously deleted all objects from the shared S3 bucket bugbash-agentcore-code-{account}-{region}.

The npm and main matrix entries are separate runners hitting the same AWS account simultaneously. Both run import-resources.test.ts. When one job's cleanup finishes first, it nuked all S3 objects — including the code.zip the other job uploaded and was still using, causing downstream resource creation failures.

The fix: each run now gets a unique RESOURCE_SUFFIX (the test's timestamp-based suffix). This scopes:

The resources file: bugbash-resources-{suffix}.json instead of bugbash-resources.json
The S3 prefix: bugbash-{suffix}/code.zip instead of bugbash/code.zip
The S3 cleanup: only deletes objects under its own prefix

Related Issue

N/A — CI performance improvement.

Documentation PR

N/A

Type of Change

Testing

Tried different shard sizes, runs are here: https://github.com/aws/agentcore-cli/actions/runs/25165632571, https://github.com/aws/agentcore-cli/actions/runs/25166733599

I ran npm run test:unit and npm run test:integ
I ran npm run typecheck
I ran npm run lint
If I modified src/assets/, I ran npm run test:update-snapshots and committed the updated snapshots

Checklist

I have read the CONTRIBUTING document
I have added any necessary tests that prove my fix is effective or my feature works
I have updated the documentation accordingly
I have added an appropriate example to the documentation to outline the feature, or no new docs are needed
My changes generate no new warnings
Any dependent changes have been merged and published

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the
terms of your choice.

agentcore-cli-automation · 2026-04-29T14:42:54Z

Reviewed — looks good to merge.

Verified:

vitest run --project e2e --shard=X/4 is a supported combination in vitest 4.x
16 e2e test files distribute across 4 shards (~4 files each); imbalance is acceptable given fail-fast: false and 60-minute timeout
Parallel test concerns look fine: cleanupStaleCredentialProviders() uses a 30-min cutoff so it won't touch resources from concurrently running shards; agent names include framework/provider prefixes so they won't collide across files
No production code touched; failure mode is just a CI run that surfaces the issue, easy to revert

One thing worth noting (not blocking): your local measurement validated 2 concurrent deploys, but this moves to 8 concurrent (4 shards × 2 cdk-sources). If you see intermittent CloudFormation/AgentCore control-plane throttling in the first few scheduled runs, that'd be the likely cause — but worth finding out empirically rather than preemptively capping.

- Add 6-way vitest sharding to the cdk-source matrix (2 → 12 parallel runners) - Isolate import test resources per run via RESOURCE_SUFFIX to prevent concurrent conflicts

Hweinstock · 2026-04-30T13:17:17Z

/strands review

github-actions · 2026-04-30T13:49:31Z

Code Review Summary

I've reviewed PR #1016, which adds 6-way vitest sharding to reduce E2E test time from 19-25 min to ~11.5 min. The approach is sound and well-measured, but there is one blocking issue that must be fixed before merge.

🔴 Blocking Issues

Missing import in `cleanup_resources.py`

File:e2e-tests/fixtures/import/cleanup_resources.py
Lines: 15, 25

The script uses RESOURCE_SUFFIX on line 25 but doesn't import it from common.py on line 15:

# Line 15 - current importsfromcommonimportREGION, RESOURCES_FILE, get_control_client, get_account_id# Line 25 - uses RESOURCE_SUFFIX (not imported!)prefix=f"bugbash-{RESOURCE_SUFFIX}/"ifRESOURCE_SUFFIXelse""

Impact: This will cause a NameError: name 'RESOURCE_SUFFIX' is not defined when cleanup runs after a test, potentially leaving resources in AWS.

Fix: Add RESOURCE_SUFFIX to the import statement:

fromcommonimportREGION, RESOURCES_FILE, RESOURCE_SUFFIX, get_control_client, get_account_id

💡 Suggestions (non-blocking)

1. Resource suffix collision risk

Files:e2e-tests/import-resources.test.ts:37, e2e-tests/import-gateway.test.ts:34

Both tests generate the suffix using Date.now():

constsuffix=Date.now().toString().slice(-8);

In a sharded run, if two shards start within the same millisecond and run the same test, they could collide on resource names. While unlikely, using a more unique identifier would be more robust:

constsuffix=`${Date.now()}-${randomUUID().slice(0,8)}`;

Or include the shard number if available in the environment.

2. S3 cleanup safety concern

File:e2e-tests/fixtures/import/cleanup_resources.py:25

When RESOURCE_SUFFIX is empty or unset, the prefix becomes "", which means list_objects_v2 will return all objects in the bucket:

prefix=f"bugbash-{RESOURCE_SUFFIX}/"ifRESOURCE_SUFFIXelse""list_args= {"Bucket": bucket_name}
ifprefix: # Empty string is falsy, so this branch is skippedlist_args["Prefix"] =prefixresp=s3.list_objects_v2(**list_args) # Lists ALL objects

This could accidentally delete objects from other concurrent runs if the env var is missing. Consider:

Making RESOURCE_SUFFIX required (fail fast if unset)
Or use a fallback like "default" instead of empty string

3. Documentation: Why 6 shards?

File:.github/workflows/e2e-tests-full.yml:30

The PR description explains the measured results for 4, 6, and 8 shards, and that 6 is optimal. Consider adding a brief comment in the workflow file explaining why 6 was chosen:

matrix:
cdk-source: [npm, main]# 6 shards provides optimal balance (~11.5 min wall-clock)# 8 shards showed no improvement due to memory deploy (274s) and container build (204s) bottlenecksshard: ['1/6', '2/6', '3/6', '4/6', '5/6', '6/6']

❓ Questions

Concurrency increase: 2 → 12 parallel deploys

File:.github/workflows/e2e-tests-full.yml:28-30

The PR moves from 2 concurrent runners (2 cdk-source entries) to 12 (2 × 6 shards). The description mentions testing 2 concurrent deploys locally but not 12.

Question: Have you verified that the AWS account's CloudFormation/AgentCore control plane can handle 12 concurrent stacks without throttling? The bot comment mentioned this could be "worth finding out empirically," but it would be good to confirm if you've already done a test run with the full matrix.

✅ Positive Aspects

Well-measured performance improvement: Clear data showing 19-25 min → 11.5 min
Proper resource isolation: Using RESOURCE_SUFFIX to scope S3 objects and resources files per run prevents conflicts between npm and main matrix entries
No production code changes: Only CI and test infrastructure, easy to revert if issues arise
Comprehensive testing: The PR description shows you tested 4, 6, and 8 shards to find the optimal configuration

Next Steps

Fix the blocking import issue in cleanup_resources.py
Consider the suggestions above (especially the S3 cleanup safety concern)
Confirm the 12 concurrent deploys won't throttle in CI

Overall, this is a great performance improvement with a solid approach. Once the import bug is fixed, this should be good to merge! 🚀

Hweinstock · 2026-04-30T16:03:49Z

Addressed comments and re ran https://github.com/aws/agentcore-cli/actions/runs/25175265881.

Hweinstock · 2026-04-29T14:40:51Z

Description

The full E2E suite takes 19-25 min because all 17 test files run on a single runner per CDK source. Each test deploys its own CloudFormation stack, and measured timings show parallel deploys have zero degradation (77s stays 77s with concurrent stacks).

This adds 6-way vitest sharding via the GitHub Actions matrix, going from 2 to 12 parallel runners. Also isolates import test resources per run to prevent conflicts when npm and main matrix entries hit the same AWS account concurrently.

Measured results:

Shards	Wall-clock	Slowest shard
1 (baseline)	19-25 min	—
4	~15 min	890s
6	~11.5 min	691s
8	~12.0 min	717s

6 shards appears to be the sweet spot. Adding more shards creates diminishing returns since the main bottleneck is the deploy (274s) and container build (204s). This is roughly what the e2e tests take on PRs when running minimal files so we're likely close to how fast we can get it.

Inspired by #989.

Docs: https://vitest.dev/guide/improving-performance.html#sharding

Import test resource isolation

The import-resources.test.ts setup creates AWS resources (runtime, memory, evaluator) via Python scripts, then saves their ARNs to bugbash-resources.json. The cleanup script deletes those resources and also calls cleanup_s3_code_objects() which previously deleted all objects from the shared S3 bucket bugbash-agentcore-code-{account}-{region}.

The npm and main matrix entries are separate runners hitting the same AWS account simultaneously. Both run import-resources.test.ts. When one job's cleanup finishes first, it nuked all S3 objects — including the code.zip the other job uploaded and was still using, causing downstream resource creation failures.

The fix: each run now gets a unique RESOURCE_SUFFIX (the test's timestamp-based suffix). This scopes:

The resources file: bugbash-resources-{suffix}.json instead of bugbash-resources.json
The S3 prefix: bugbash-{suffix}/code.zip instead of bugbash/code.zip
The S3 cleanup: only deletes objects under its own prefix

Related Issue

N/A — CI performance improvement.

Documentation PR

N/A

Type of Change

Testing

Tried different shard sizes, runs are here: https://github.com/aws/agentcore-cli/actions/runs/25165632571, https://github.com/aws/agentcore-cli/actions/runs/25166733599

I ran npm run test:unit and npm run test:integ
I ran npm run typecheck
I ran npm run lint
If I modified src/assets/, I ran npm run test:update-snapshots and committed the updated snapshots

Checklist

I have read the CONTRIBUTING document
I have added any necessary tests that prove my fix is effective or my feature works
I have updated the documentation accordingly
I have added an appropriate example to the documentation to outline the feature, or no new docs are needed
My changes generate no new warnings
Any dependent changes have been merged and published

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the
terms of your choice.

agentcore-cli-automation · 2026-04-29T14:42:54Z

Reviewed — looks good to merge.

Verified:

vitest run --project e2e --shard=X/4 is a supported combination in vitest 4.x
16 e2e test files distribute across 4 shards (~4 files each); imbalance is acceptable given fail-fast: false and 60-minute timeout
Parallel test concerns look fine: cleanupStaleCredentialProviders() uses a 30-min cutoff so it won't touch resources from concurrently running shards; agent names include framework/provider prefixes so they won't collide across files
No production code touched; failure mode is just a CI run that surfaces the issue, easy to revert

One thing worth noting (not blocking): your local measurement validated 2 concurrent deploys, but this moves to 8 concurrent (4 shards × 2 cdk-sources). If you see intermittent CloudFormation/AgentCore control-plane throttling in the first few scheduled runs, that'd be the likely cause — but worth finding out empirically rather than preemptively capping.

- Add 6-way vitest sharding to the cdk-source matrix (2 → 12 parallel runners) - Isolate import test resources per run via RESOURCE_SUFFIX to prevent concurrent conflicts

Hweinstock · 2026-04-30T13:17:17Z

/strands review

github-actions · 2026-04-30T13:49:31Z

Code Review Summary

I've reviewed PR #1016, which adds 6-way vitest sharding to reduce E2E test time from 19-25 min to ~11.5 min. The approach is sound and well-measured, but there is one blocking issue that must be fixed before merge.

🔴 Blocking Issues

Missing import in `cleanup_resources.py`

File:e2e-tests/fixtures/import/cleanup_resources.py
Lines: 15, 25

The script uses RESOURCE_SUFFIX on line 25 but doesn't import it from common.py on line 15:

# Line 15 - current importsfromcommonimportREGION, RESOURCES_FILE, get_control_client, get_account_id# Line 25 - uses RESOURCE_SUFFIX (not imported!)prefix=f"bugbash-{RESOURCE_SUFFIX}/"ifRESOURCE_SUFFIXelse""

Impact: This will cause a NameError: name 'RESOURCE_SUFFIX' is not defined when cleanup runs after a test, potentially leaving resources in AWS.

Fix: Add RESOURCE_SUFFIX to the import statement:

fromcommonimportREGION, RESOURCES_FILE, RESOURCE_SUFFIX, get_control_client, get_account_id

💡 Suggestions (non-blocking)

1. Resource suffix collision risk

Files:e2e-tests/import-resources.test.ts:37, e2e-tests/import-gateway.test.ts:34

Both tests generate the suffix using Date.now():

constsuffix=Date.now().toString().slice(-8);

In a sharded run, if two shards start within the same millisecond and run the same test, they could collide on resource names. While unlikely, using a more unique identifier would be more robust:

constsuffix=`${Date.now()}-${randomUUID().slice(0,8)}`;

Or include the shard number if available in the environment.

2. S3 cleanup safety concern

File:e2e-tests/fixtures/import/cleanup_resources.py:25

When RESOURCE_SUFFIX is empty or unset, the prefix becomes "", which means list_objects_v2 will return all objects in the bucket:

prefix=f"bugbash-{RESOURCE_SUFFIX}/"ifRESOURCE_SUFFIXelse""list_args= {"Bucket": bucket_name}
ifprefix: # Empty string is falsy, so this branch is skippedlist_args["Prefix"] =prefixresp=s3.list_objects_v2(**list_args) # Lists ALL objects

This could accidentally delete objects from other concurrent runs if the env var is missing. Consider:

Making RESOURCE_SUFFIX required (fail fast if unset)
Or use a fallback like "default" instead of empty string

3. Documentation: Why 6 shards?

File:.github/workflows/e2e-tests-full.yml:30

The PR description explains the measured results for 4, 6, and 8 shards, and that 6 is optimal. Consider adding a brief comment in the workflow file explaining why 6 was chosen:

matrix:
cdk-source: [npm, main]# 6 shards provides optimal balance (~11.5 min wall-clock)# 8 shards showed no improvement due to memory deploy (274s) and container build (204s) bottlenecksshard: ['1/6', '2/6', '3/6', '4/6', '5/6', '6/6']

❓ Questions

Concurrency increase: 2 → 12 parallel deploys

File:.github/workflows/e2e-tests-full.yml:28-30

The PR moves from 2 concurrent runners (2 cdk-source entries) to 12 (2 × 6 shards). The description mentions testing 2 concurrent deploys locally but not 12.

Question: Have you verified that the AWS account's CloudFormation/AgentCore control plane can handle 12 concurrent stacks without throttling? The bot comment mentioned this could be "worth finding out empirically," but it would be good to confirm if you've already done a test run with the full matrix.

✅ Positive Aspects

Well-measured performance improvement: Clear data showing 19-25 min → 11.5 min
Proper resource isolation: Using RESOURCE_SUFFIX to scope S3 objects and resources files per run prevents conflicts between npm and main matrix entries
No production code changes: Only CI and test infrastructure, easy to revert if issues arise
Comprehensive testing: The PR description shows you tested 4, 6, and 8 shards to find the optimal configuration

Next Steps

Fix the blocking import issue in cleanup_resources.py
Consider the suggestions above (especially the S3 cleanup safety concern)
Confirm the 12 concurrent deploys won't throttle in CI

Overall, this is a great performance improvement with a solid approach. Once the import bug is fixed, this should be good to merge! 🚀

Hweinstock · 2026-04-30T16:03:49Z

Addressed comments and re ran https://github.com/aws/agentcore-cli/actions/runs/25175265881.

Hweinstock · 2026-04-29T14:40:51Z

Description

The full E2E suite takes 19-25 min because all 17 test files run on a single runner per CDK source. Each test deploys its own CloudFormation stack, and measured timings show parallel deploys have zero degradation (77s stays 77s with concurrent stacks).

This adds 6-way vitest sharding via the GitHub Actions matrix, going from 2 to 12 parallel runners. Also isolates import test resources per run to prevent conflicts when npm and main matrix entries hit the same AWS account concurrently.

Measured results:

Shards	Wall-clock	Slowest shard
1 (baseline)	19-25 min	—
4	~15 min	890s
6	~11.5 min	691s
8	~12.0 min	717s

6 shards appears to be the sweet spot. Adding more shards creates diminishing returns since the main bottleneck is the deploy (274s) and container build (204s). This is roughly what the e2e tests take on PRs when running minimal files so we're likely close to how fast we can get it.

Inspired by #989.

Docs: https://vitest.dev/guide/improving-performance.html#sharding

Import test resource isolation

The import-resources.test.ts setup creates AWS resources (runtime, memory, evaluator) via Python scripts, then saves their ARNs to bugbash-resources.json. The cleanup script deletes those resources and also calls cleanup_s3_code_objects() which previously deleted all objects from the shared S3 bucket bugbash-agentcore-code-{account}-{region}.

The npm and main matrix entries are separate runners hitting the same AWS account simultaneously. Both run import-resources.test.ts. When one job's cleanup finishes first, it nuked all S3 objects — including the code.zip the other job uploaded and was still using, causing downstream resource creation failures.

The fix: each run now gets a unique RESOURCE_SUFFIX (the test's timestamp-based suffix). This scopes:

The resources file: bugbash-resources-{suffix}.json instead of bugbash-resources.json
The S3 prefix: bugbash-{suffix}/code.zip instead of bugbash/code.zip
The S3 cleanup: only deletes objects under its own prefix

Related Issue

N/A — CI performance improvement.

Documentation PR

N/A

Type of Change

Testing

Tried different shard sizes, runs are here: https://github.com/aws/agentcore-cli/actions/runs/25165632571, https://github.com/aws/agentcore-cli/actions/runs/25166733599

I ran npm run test:unit and npm run test:integ
I ran npm run typecheck
I ran npm run lint
If I modified src/assets/, I ran npm run test:update-snapshots and committed the updated snapshots

Checklist

I have read the CONTRIBUTING document
I have added any necessary tests that prove my fix is effective or my feature works
I have updated the documentation accordingly
I have added an appropriate example to the documentation to outline the feature, or no new docs are needed
My changes generate no new warnings
Any dependent changes have been merged and published

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the
terms of your choice.

agentcore-cli-automation · 2026-04-29T14:42:54Z

Reviewed — looks good to merge.

Verified:

vitest run --project e2e --shard=X/4 is a supported combination in vitest 4.x
16 e2e test files distribute across 4 shards (~4 files each); imbalance is acceptable given fail-fast: false and 60-minute timeout
Parallel test concerns look fine: cleanupStaleCredentialProviders() uses a 30-min cutoff so it won't touch resources from concurrently running shards; agent names include framework/provider prefixes so they won't collide across files
No production code touched; failure mode is just a CI run that surfaces the issue, easy to revert

One thing worth noting (not blocking): your local measurement validated 2 concurrent deploys, but this moves to 8 concurrent (4 shards × 2 cdk-sources). If you see intermittent CloudFormation/AgentCore control-plane throttling in the first few scheduled runs, that'd be the likely cause — but worth finding out empirically rather than preemptively capping.

- Add 6-way vitest sharding to the cdk-source matrix (2 → 12 parallel runners) - Isolate import test resources per run via RESOURCE_SUFFIX to prevent concurrent conflicts

Hweinstock · 2026-04-30T13:17:17Z

/strands review

github-actions · 2026-04-30T13:49:31Z

Code Review Summary

I've reviewed PR #1016, which adds 6-way vitest sharding to reduce E2E test time from 19-25 min to ~11.5 min. The approach is sound and well-measured, but there is one blocking issue that must be fixed before merge.

🔴 Blocking Issues

Missing import in `cleanup_resources.py`

File:e2e-tests/fixtures/import/cleanup_resources.py
Lines: 15, 25

The script uses RESOURCE_SUFFIX on line 25 but doesn't import it from common.py on line 15:

# Line 15 - current importsfromcommonimportREGION, RESOURCES_FILE, get_control_client, get_account_id# Line 25 - uses RESOURCE_SUFFIX (not imported!)prefix=f"bugbash-{RESOURCE_SUFFIX}/"ifRESOURCE_SUFFIXelse""

Impact: This will cause a NameError: name 'RESOURCE_SUFFIX' is not defined when cleanup runs after a test, potentially leaving resources in AWS.

Fix: Add RESOURCE_SUFFIX to the import statement:

fromcommonimportREGION, RESOURCES_FILE, RESOURCE_SUFFIX, get_control_client, get_account_id

💡 Suggestions (non-blocking)

1. Resource suffix collision risk

Files:e2e-tests/import-resources.test.ts:37, e2e-tests/import-gateway.test.ts:34

Both tests generate the suffix using Date.now():

constsuffix=Date.now().toString().slice(-8);

In a sharded run, if two shards start within the same millisecond and run the same test, they could collide on resource names. While unlikely, using a more unique identifier would be more robust:

constsuffix=`${Date.now()}-${randomUUID().slice(0,8)}`;

Or include the shard number if available in the environment.

2. S3 cleanup safety concern

File:e2e-tests/fixtures/import/cleanup_resources.py:25

When RESOURCE_SUFFIX is empty or unset, the prefix becomes "", which means list_objects_v2 will return all objects in the bucket:

prefix=f"bugbash-{RESOURCE_SUFFIX}/"ifRESOURCE_SUFFIXelse""list_args= {"Bucket": bucket_name}
ifprefix: # Empty string is falsy, so this branch is skippedlist_args["Prefix"] =prefixresp=s3.list_objects_v2(**list_args) # Lists ALL objects

This could accidentally delete objects from other concurrent runs if the env var is missing. Consider:

Making RESOURCE_SUFFIX required (fail fast if unset)
Or use a fallback like "default" instead of empty string

3. Documentation: Why 6 shards?

File:.github/workflows/e2e-tests-full.yml:30

The PR description explains the measured results for 4, 6, and 8 shards, and that 6 is optimal. Consider adding a brief comment in the workflow file explaining why 6 was chosen:

matrix:
cdk-source: [npm, main]# 6 shards provides optimal balance (~11.5 min wall-clock)# 8 shards showed no improvement due to memory deploy (274s) and container build (204s) bottlenecksshard: ['1/6', '2/6', '3/6', '4/6', '5/6', '6/6']

❓ Questions

Concurrency increase: 2 → 12 parallel deploys

File:.github/workflows/e2e-tests-full.yml:28-30

The PR moves from 2 concurrent runners (2 cdk-source entries) to 12 (2 × 6 shards). The description mentions testing 2 concurrent deploys locally but not 12.

Question: Have you verified that the AWS account's CloudFormation/AgentCore control plane can handle 12 concurrent stacks without throttling? The bot comment mentioned this could be "worth finding out empirically," but it would be good to confirm if you've already done a test run with the full matrix.

✅ Positive Aspects

Well-measured performance improvement: Clear data showing 19-25 min → 11.5 min
Proper resource isolation: Using RESOURCE_SUFFIX to scope S3 objects and resources files per run prevents conflicts between npm and main matrix entries
No production code changes: Only CI and test infrastructure, easy to revert if issues arise
Comprehensive testing: The PR description shows you tested 4, 6, and 8 shards to find the optimal configuration

Next Steps

Fix the blocking import issue in cleanup_resources.py
Consider the suggestions above (especially the S3 cleanup safety concern)
Confirm the 12 concurrent deploys won't throttle in CI

Overall, this is a great performance improvement with a solid approach. Once the import bug is fixed, this should be good to merge! 🚀

Hweinstock · 2026-04-30T16:03:49Z

Addressed comments and re ran https://github.com/aws/agentcore-cli/actions/runs/25175265881.

Hweinstock · 2026-04-29T14:40:51Z

Description

The full E2E suite takes 19-25 min because all 17 test files run on a single runner per CDK source. Each test deploys its own CloudFormation stack, and measured timings show parallel deploys have zero degradation (77s stays 77s with concurrent stacks).

This adds 6-way vitest sharding via the GitHub Actions matrix, going from 2 to 12 parallel runners. Also isolates import test resources per run to prevent conflicts when npm and main matrix entries hit the same AWS account concurrently.

Measured results:

Shards	Wall-clock	Slowest shard
1 (baseline)	19-25 min	—
4	~15 min	890s
6	~11.5 min	691s
8	~12.0 min	717s

6 shards appears to be the sweet spot. Adding more shards creates diminishing returns since the main bottleneck is the deploy (274s) and container build (204s). This is roughly what the e2e tests take on PRs when running minimal files so we're likely close to how fast we can get it.

Inspired by #989.

Docs: https://vitest.dev/guide/improving-performance.html#sharding

Import test resource isolation

The import-resources.test.ts setup creates AWS resources (runtime, memory, evaluator) via Python scripts, then saves their ARNs to bugbash-resources.json. The cleanup script deletes those resources and also calls cleanup_s3_code_objects() which previously deleted all objects from the shared S3 bucket bugbash-agentcore-code-{account}-{region}.

The npm and main matrix entries are separate runners hitting the same AWS account simultaneously. Both run import-resources.test.ts. When one job's cleanup finishes first, it nuked all S3 objects — including the code.zip the other job uploaded and was still using, causing downstream resource creation failures.

The fix: each run now gets a unique RESOURCE_SUFFIX (the test's timestamp-based suffix). This scopes:

The resources file: bugbash-resources-{suffix}.json instead of bugbash-resources.json
The S3 prefix: bugbash-{suffix}/code.zip instead of bugbash/code.zip
The S3 cleanup: only deletes objects under its own prefix

Related Issue

N/A — CI performance improvement.

Documentation PR

N/A

Type of Change

Testing

Tried different shard sizes, runs are here: https://github.com/aws/agentcore-cli/actions/runs/25165632571, https://github.com/aws/agentcore-cli/actions/runs/25166733599

I ran npm run test:unit and npm run test:integ
I ran npm run typecheck
I ran npm run lint
If I modified src/assets/, I ran npm run test:update-snapshots and committed the updated snapshots

Checklist

I have read the CONTRIBUTING document
I have added any necessary tests that prove my fix is effective or my feature works
I have updated the documentation accordingly
I have added an appropriate example to the documentation to outline the feature, or no new docs are needed
My changes generate no new warnings
Any dependent changes have been merged and published

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the
terms of your choice.

agentcore-cli-automation · 2026-04-29T14:42:54Z

Reviewed — looks good to merge.

Verified:

vitest run --project e2e --shard=X/4 is a supported combination in vitest 4.x
16 e2e test files distribute across 4 shards (~4 files each); imbalance is acceptable given fail-fast: false and 60-minute timeout
Parallel test concerns look fine: cleanupStaleCredentialProviders() uses a 30-min cutoff so it won't touch resources from concurrently running shards; agent names include framework/provider prefixes so they won't collide across files
No production code touched; failure mode is just a CI run that surfaces the issue, easy to revert

One thing worth noting (not blocking): your local measurement validated 2 concurrent deploys, but this moves to 8 concurrent (4 shards × 2 cdk-sources). If you see intermittent CloudFormation/AgentCore control-plane throttling in the first few scheduled runs, that'd be the likely cause — but worth finding out empirically rather than preemptively capping.

- Add 6-way vitest sharding to the cdk-source matrix (2 → 12 parallel runners) - Isolate import test resources per run via RESOURCE_SUFFIX to prevent concurrent conflicts

Hweinstock · 2026-04-30T13:17:17Z

/strands review

github-actions · 2026-04-30T13:49:31Z

Code Review Summary

I've reviewed PR #1016, which adds 6-way vitest sharding to reduce E2E test time from 19-25 min to ~11.5 min. The approach is sound and well-measured, but there is one blocking issue that must be fixed before merge.

🔴 Blocking Issues

Missing import in `cleanup_resources.py`

File:e2e-tests/fixtures/import/cleanup_resources.py
Lines: 15, 25

The script uses RESOURCE_SUFFIX on line 25 but doesn't import it from common.py on line 15:

# Line 15 - current importsfromcommonimportREGION, RESOURCES_FILE, get_control_client, get_account_id# Line 25 - uses RESOURCE_SUFFIX (not imported!)prefix=f"bugbash-{RESOURCE_SUFFIX}/"ifRESOURCE_SUFFIXelse""

Impact: This will cause a NameError: name 'RESOURCE_SUFFIX' is not defined when cleanup runs after a test, potentially leaving resources in AWS.

Fix: Add RESOURCE_SUFFIX to the import statement:

fromcommonimportREGION, RESOURCES_FILE, RESOURCE_SUFFIX, get_control_client, get_account_id

💡 Suggestions (non-blocking)

1. Resource suffix collision risk

Files:e2e-tests/import-resources.test.ts:37, e2e-tests/import-gateway.test.ts:34

Both tests generate the suffix using Date.now():

constsuffix=Date.now().toString().slice(-8);

In a sharded run, if two shards start within the same millisecond and run the same test, they could collide on resource names. While unlikely, using a more unique identifier would be more robust:

constsuffix=`${Date.now()}-${randomUUID().slice(0,8)}`;

Or include the shard number if available in the environment.

2. S3 cleanup safety concern

File:e2e-tests/fixtures/import/cleanup_resources.py:25

When RESOURCE_SUFFIX is empty or unset, the prefix becomes "", which means list_objects_v2 will return all objects in the bucket:

prefix=f"bugbash-{RESOURCE_SUFFIX}/"ifRESOURCE_SUFFIXelse""list_args= {"Bucket": bucket_name}
ifprefix: # Empty string is falsy, so this branch is skippedlist_args["Prefix"] =prefixresp=s3.list_objects_v2(**list_args) # Lists ALL objects

This could accidentally delete objects from other concurrent runs if the env var is missing. Consider:

Making RESOURCE_SUFFIX required (fail fast if unset)
Or use a fallback like "default" instead of empty string

3. Documentation: Why 6 shards?

File:.github/workflows/e2e-tests-full.yml:30

The PR description explains the measured results for 4, 6, and 8 shards, and that 6 is optimal. Consider adding a brief comment in the workflow file explaining why 6 was chosen:

matrix:
cdk-source: [npm, main]# 6 shards provides optimal balance (~11.5 min wall-clock)# 8 shards showed no improvement due to memory deploy (274s) and container build (204s) bottlenecksshard: ['1/6', '2/6', '3/6', '4/6', '5/6', '6/6']

❓ Questions

Concurrency increase: 2 → 12 parallel deploys

File:.github/workflows/e2e-tests-full.yml:28-30

The PR moves from 2 concurrent runners (2 cdk-source entries) to 12 (2 × 6 shards). The description mentions testing 2 concurrent deploys locally but not 12.

Question: Have you verified that the AWS account's CloudFormation/AgentCore control plane can handle 12 concurrent stacks without throttling? The bot comment mentioned this could be "worth finding out empirically," but it would be good to confirm if you've already done a test run with the full matrix.

✅ Positive Aspects

Well-measured performance improvement: Clear data showing 19-25 min → 11.5 min
Proper resource isolation: Using RESOURCE_SUFFIX to scope S3 objects and resources files per run prevents conflicts between npm and main matrix entries
No production code changes: Only CI and test infrastructure, easy to revert if issues arise
Comprehensive testing: The PR description shows you tested 4, 6, and 8 shards to find the optimal configuration

Next Steps

Fix the blocking import issue in cleanup_resources.py
Consider the suggestions above (especially the S3 cleanup safety concern)
Confirm the 12 concurrent deploys won't throttle in CI

Overall, this is a great performance improvement with a solid approach. Once the import bug is fixed, this should be good to merge! 🚀

Hweinstock · 2026-04-30T16:03:49Z

Addressed comments and re ran https://github.com/aws/agentcore-cli/actions/runs/25175265881.

Hweinstock · 2026-04-29T14:40:51Z

Description

The full E2E suite takes 19-25 min because all 17 test files run on a single runner per CDK source. Each test deploys its own CloudFormation stack, and measured timings show parallel deploys have zero degradation (77s stays 77s with concurrent stacks).

This adds 6-way vitest sharding via the GitHub Actions matrix, going from 2 to 12 parallel runners. Also isolates import test resources per run to prevent conflicts when npm and main matrix entries hit the same AWS account concurrently.

Measured results:

Shards	Wall-clock	Slowest shard
1 (baseline)	19-25 min	—
4	~15 min	890s
6	~11.5 min	691s
8	~12.0 min	717s

6 shards appears to be the sweet spot. Adding more shards creates diminishing returns since the main bottleneck is the deploy (274s) and container build (204s). This is roughly what the e2e tests take on PRs when running minimal files so we're likely close to how fast we can get it.

Inspired by #989.

Docs: https://vitest.dev/guide/improving-performance.html#sharding

Import test resource isolation

The import-resources.test.ts setup creates AWS resources (runtime, memory, evaluator) via Python scripts, then saves their ARNs to bugbash-resources.json. The cleanup script deletes those resources and also calls cleanup_s3_code_objects() which previously deleted all objects from the shared S3 bucket bugbash-agentcore-code-{account}-{region}.

The npm and main matrix entries are separate runners hitting the same AWS account simultaneously. Both run import-resources.test.ts. When one job's cleanup finishes first, it nuked all S3 objects — including the code.zip the other job uploaded and was still using, causing downstream resource creation failures.

The fix: each run now gets a unique RESOURCE_SUFFIX (the test's timestamp-based suffix). This scopes:

The resources file: bugbash-resources-{suffix}.json instead of bugbash-resources.json
The S3 prefix: bugbash-{suffix}/code.zip instead of bugbash/code.zip
The S3 cleanup: only deletes objects under its own prefix

Related Issue

N/A — CI performance improvement.

Documentation PR

N/A

Type of Change

Testing

Tried different shard sizes, runs are here: https://github.com/aws/agentcore-cli/actions/runs/25165632571, https://github.com/aws/agentcore-cli/actions/runs/25166733599

I ran npm run test:unit and npm run test:integ
I ran npm run typecheck
I ran npm run lint
If I modified src/assets/, I ran npm run test:update-snapshots and committed the updated snapshots

Checklist

I have read the CONTRIBUTING document
I have added any necessary tests that prove my fix is effective or my feature works
I have updated the documentation accordingly
I have added an appropriate example to the documentation to outline the feature, or no new docs are needed
My changes generate no new warnings
Any dependent changes have been merged and published

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the
terms of your choice.

agentcore-cli-automation · 2026-04-29T14:42:54Z

Reviewed — looks good to merge.

Verified:

vitest run --project e2e --shard=X/4 is a supported combination in vitest 4.x
16 e2e test files distribute across 4 shards (~4 files each); imbalance is acceptable given fail-fast: false and 60-minute timeout
Parallel test concerns look fine: cleanupStaleCredentialProviders() uses a 30-min cutoff so it won't touch resources from concurrently running shards; agent names include framework/provider prefixes so they won't collide across files
No production code touched; failure mode is just a CI run that surfaces the issue, easy to revert

One thing worth noting (not blocking): your local measurement validated 2 concurrent deploys, but this moves to 8 concurrent (4 shards × 2 cdk-sources). If you see intermittent CloudFormation/AgentCore control-plane throttling in the first few scheduled runs, that'd be the likely cause — but worth finding out empirically rather than preemptively capping.

- Add 6-way vitest sharding to the cdk-source matrix (2 → 12 parallel runners) - Isolate import test resources per run via RESOURCE_SUFFIX to prevent concurrent conflicts

Hweinstock · 2026-04-30T13:17:17Z

/strands review

github-actions · 2026-04-30T13:49:31Z

Code Review Summary

I've reviewed PR #1016, which adds 6-way vitest sharding to reduce E2E test time from 19-25 min to ~11.5 min. The approach is sound and well-measured, but there is one blocking issue that must be fixed before merge.

🔴 Blocking Issues

Missing import in `cleanup_resources.py`

File:e2e-tests/fixtures/import/cleanup_resources.py
Lines: 15, 25

The script uses RESOURCE_SUFFIX on line 25 but doesn't import it from common.py on line 15:

# Line 15 - current importsfromcommonimportREGION, RESOURCES_FILE, get_control_client, get_account_id# Line 25 - uses RESOURCE_SUFFIX (not imported!)prefix=f"bugbash-{RESOURCE_SUFFIX}/"ifRESOURCE_SUFFIXelse""

Impact: This will cause a NameError: name 'RESOURCE_SUFFIX' is not defined when cleanup runs after a test, potentially leaving resources in AWS.

Fix: Add RESOURCE_SUFFIX to the import statement:

fromcommonimportREGION, RESOURCES_FILE, RESOURCE_SUFFIX, get_control_client, get_account_id

💡 Suggestions (non-blocking)

1. Resource suffix collision risk

Files:e2e-tests/import-resources.test.ts:37, e2e-tests/import-gateway.test.ts:34

Both tests generate the suffix using Date.now():

constsuffix=Date.now().toString().slice(-8);

In a sharded run, if two shards start within the same millisecond and run the same test, they could collide on resource names. While unlikely, using a more unique identifier would be more robust:

constsuffix=`${Date.now()}-${randomUUID().slice(0,8)}`;

Or include the shard number if available in the environment.

2. S3 cleanup safety concern

File:e2e-tests/fixtures/import/cleanup_resources.py:25

When RESOURCE_SUFFIX is empty or unset, the prefix becomes "", which means list_objects_v2 will return all objects in the bucket:

prefix=f"bugbash-{RESOURCE_SUFFIX}/"ifRESOURCE_SUFFIXelse""list_args= {"Bucket": bucket_name}
ifprefix: # Empty string is falsy, so this branch is skippedlist_args["Prefix"] =prefixresp=s3.list_objects_v2(**list_args) # Lists ALL objects

This could accidentally delete objects from other concurrent runs if the env var is missing. Consider:

Making RESOURCE_SUFFIX required (fail fast if unset)
Or use a fallback like "default" instead of empty string

3. Documentation: Why 6 shards?

File:.github/workflows/e2e-tests-full.yml:30

The PR description explains the measured results for 4, 6, and 8 shards, and that 6 is optimal. Consider adding a brief comment in the workflow file explaining why 6 was chosen:

matrix:
cdk-source: [npm, main]# 6 shards provides optimal balance (~11.5 min wall-clock)# 8 shards showed no improvement due to memory deploy (274s) and container build (204s) bottlenecksshard: ['1/6', '2/6', '3/6', '4/6', '5/6', '6/6']

❓ Questions

Concurrency increase: 2 → 12 parallel deploys

File:.github/workflows/e2e-tests-full.yml:28-30

The PR moves from 2 concurrent runners (2 cdk-source entries) to 12 (2 × 6 shards). The description mentions testing 2 concurrent deploys locally but not 12.

Question: Have you verified that the AWS account's CloudFormation/AgentCore control plane can handle 12 concurrent stacks without throttling? The bot comment mentioned this could be "worth finding out empirically," but it would be good to confirm if you've already done a test run with the full matrix.

✅ Positive Aspects

Well-measured performance improvement: Clear data showing 19-25 min → 11.5 min
Proper resource isolation: Using RESOURCE_SUFFIX to scope S3 objects and resources files per run prevents conflicts between npm and main matrix entries
No production code changes: Only CI and test infrastructure, easy to revert if issues arise
Comprehensive testing: The PR description shows you tested 4, 6, and 8 shards to find the optimal configuration

Next Steps

Fix the blocking import issue in cleanup_resources.py
Consider the suggestions above (especially the S3 cleanup safety concern)
Confirm the 12 concurrent deploys won't throttle in CI

Overall, this is a great performance improvement with a solid approach. Once the import bug is fixed, this should be good to merge! 🚀

Hweinstock · 2026-04-30T16:03:49Z

Addressed comments and re ran https://github.com/aws/agentcore-cli/actions/runs/25175265881.

github-actionsBot added size/xs PR size: XS agentcore-harness-reviewing AgentCore Harness review in progress labels Apr 29, 2026

Hweinstock temporarily deployed to e2e-testing April 29, 2026 14:42 — with GitHub Actions Inactive

github-actionsBot removed agentcore-harness-reviewing AgentCore Harness review in progress size/xs PR size: XS labels Apr 29, 2026

Hweinstock temporarily deployed to e2e-testing April 29, 2026 17:31 — with GitHub Actions Inactive

Hweinstock had a problem deploying to e2e-testing April 29, 2026 17:31 — with GitHub Actions Failure

github-actionsBot added the size/s PR size: S label Apr 29, 2026

Hweinstock temporarily deployed to e2e-testing April 29, 2026 18:03 — with GitHub Actions Inactive

Hweinstock temporarily deployed to e2e-testing April 29, 2026 19:08 — with GitHub Actions Inactive

Hweinstock temporarily deployed to e2e-testing April 29, 2026 20:23 — with GitHub Actions Inactive

Hweinstock had a problem deploying to e2e-testing April 29, 2026 20:23 — with GitHub Actions Failure

github-actionsBot added size/s PR size: S and removed size/s PR size: S labels Apr 30, 2026

Hweinstock temporarily deployed to e2e-testing April 30, 2026 12:34 — with GitHub Actions Inactive

Hweinstock temporarily deployed to e2e-testing April 30, 2026 12:59 — with GitHub Actions Inactive

ci: shard e2e full suite across 6 runners
df22788
- Add 6-way vitest sharding to the cdk-source matrix (2 → 12 parallel runners) - Isolate import test resources per run via RESOURCE_SUFFIX to prevent concurrent conflicts

Hweinstock force-pushed the ci/e2e-shard branch from fe5b107 to df22788Compare April 30, 2026 13:15

github-actionsBot added size/s PR size: S and removed size/s PR size: S labels Apr 30, 2026

github-actionsBot added the strands-running label Apr 30, 2026

Hweinstock changed the title ~~ci: cut e2e time in half via vitest sharding~~ci: cut full e2e time in half via vitest shardingApr 30, 2026

Hweinstock marked this pull request as ready for review April 30, 2026 13:36

Hweinstock requested a review from a teamApril 30, 2026 13:36

tejaskash previously approved these changes Apr 30, 2026
View reviewed changes

github-actionsBot removed the strands-running label Apr 30, 2026

fix: import RESOURCE_SUFFIX in cleanup_resources.py
28c60db

Hweinstock dismissed tejaskash’s stale review via 28c60dbApril 30, 2026 15:46

tejaskash approved these changes Apr 30, 2026
View reviewed changes

This was referenced Apr 30, 2026
chore: merge main into preview manually due to workflow changes #1064
Closed
chore: merge main into preview manually due to workflow changes. #1063
Closed