feat(tests): add E2E and integration tests for Harness primitive - #1021

Merged
tejaskash merged 4 commits into
previewfrom
feat/harness-e2e-integ-tests
Apr 29, 2026
Merged

feat(tests): add E2E and integration tests for Harness primitive#1021
tejaskash merged 4 commits into
previewfrom
feat/harness-e2e-integ-tests

Conversation

@tejaskash

@tejaskashtejaskash commented Apr 29, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Adds E2E tests for Harness primitive across all three model providers (Bedrock, OpenAI, Gemini), each covering create → deploy → invoke → status → teardown against real AWS
  • Adds integration tests (15 cases) covering add/remove lifecycle, config options (truncation, lifecycle, no-memory, non-Bedrock providers), validation errors, and project-level create-with-harness
  • Adds harness-bedrock to the PR baseline E2E run in CI
  • OpenAI and Gemini E2E tests skip memory (--no-harness-memory) to reduce deploy time; Bedrock keeps memory for full coverage

Design

Uses a factory pattern (createHarnessE2ESuite) mirroring the existing agent E2E helper. Each provider gets its own test file so vitest runs all three in parallel. Tests within each file are sequential since they depend on prior steps (deploy before invoke before status).

Test plan

  • All 15 integ tests pass (npm run test:integ -- add-remove-harness)
  • All 9 E2E tests pass against real AWS (3 providers × 3 tests each)
  • CI workflow includes harness-bedrock in PR baseline
  • CI passes on this PR

@tejaskash
feat(tests): add E2E and integ tests for Harness primitive
7335079
E2E tests verify create/deploy/invoke/status against real AWS for
Bedrock, OpenAI, and Gemini providers using a factory pattern that
mirrors the existing agent E2E suite. Integ tests cover the
add/remove lifecycle, configuration options (truncation, lifecycle,
model provider), validation errors, and project scaffolding.
Harness-bedrock is added to the PR E2E baseline alongside
strands-bedrock so every PR exercises the harness deploy path.
@tejaskash
skip memory for OpenAI/Gemini E2E harness tests to reduce deploy time
9a572eb
@github-actionsgithub-actionsBot added the agentcore-harness-reviewing AgentCore Harness review in progress label Apr 29, 2026
@agentcore-cli-automation

agentcore-cli-automation commented Apr 29, 2026

Copy link
Copy Markdown

Missing CI secrets for OpenAI/Gemini harness E2E tests

The harness-openai.test.ts and harness-gemini.test.ts suites gate on env vars OPENAI_API_KEY_ARN and GEMINI_API_KEY_ARN respectively (via skipIf(!canRun) where canRun requires the env var to be truthy). These env vars are not plumbed through .github/workflows/e2e-tests.yml — only OPENAI_API_KEY and GEMINI_API_KEY (raw keys) are set today, and those are used by the existing agent tests.

Net effect:

  • When this PR lands and a future PR touches e2e-tests/harness-openai.test.ts or e2e-tests/harness-gemini.test.ts, the file will be picked up by the changed-files detection and vitest will load the suite, but every it inside will be silently skipped. CI will report green with zero assertions run — a false-positive signal.
  • The PR description claims "All 9 E2E tests pass against real AWS (3 providers × 3 tests each)", but only the 3 Bedrock tests will actually execute in CI.

Options:

  1. Provision OPENAI_API_KEY_ARN / GEMINI_API_KEY_ARN in the E2E secret bundle (Secrets Manager ARNs pointing at existing key secrets) and add them to the env: block of the Run E2E tests step. This is the "real" fix and matches the PR's stated test plan.
  2. Drop the harness-openai.test.ts / harness-gemini.test.ts files from this PR until the ARN secrets are wired up. The createHarnessE2ESuite factory can stay for later reuse.
  3. If the env-var name is the blocker, derive the ARN at test time from the existing raw key (e.g. create an ephemeral Secrets Manager secret in beforeAll and tear it down in afterAll). This keeps the CI secret surface unchanged but adds non-trivial test plumbing.

At minimum, if option 1 isn't done before merge, the tests will look like they're running but aren't — which defeats the purpose of adding them.

@github-actionsgithub-actionsBot removed the agentcore-harness-reviewing AgentCore Harness review in progress label Apr 29, 2026
Hweinstock
Hweinstock previously approved these changes Apr 29, 2026
View reviewed changes

@HweinstockHweinstock left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice!

Comment threadinteg-tests/add-remove-harness.test.ts Outdated
@tejaskash
fix: use shared readProjectConfig from test-utils
191f80a
Address review comment — reuse existing config-reader utility instead of
inline helper.
@tejaskash
tejaskash requested a review from a teamApril 29, 2026 19:02
@tejaskash

tejaskash commented Apr 29, 2026

Copy link
Copy Markdown
ContributorAuthor

Addressed @Hweinstock's review comment: replaced inline readProjectConfig with the shared utility from src/test-utils/config-reader.ts (191f80a).

@github-actions

github-actionsBot commented Apr 29, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

Coverage Report

StatusCategoryPercentageCovered / Total
🔵Lines43.43%8702 / 20036
🔵Statements42.77%9213 / 21537
🔵Functions40.95%1468 / 3584
🔵Branches40.67%5678 / 13960
Generated in workflow #2139 for commit e8bc7bb by the Vitest Coverage Report Action

@tejaskash
fix: add non-null assertion to fix typecheck failure
e8bc7bb
The Zod-validated return type from readProjectConfig makes the find()
result possibly undefined. The preceding expect() guards against it at
runtime.
@tejaskash
tejaskash merged commit 626872f into previewApr 29, 2026
15 checks passed
@tejaskash
tejaskash deleted the feat/harness-e2e-integ-tests branch April 29, 2026 19:34
@agentcore-cli-automationagentcore-cli-automation mentioned this pull request Apr 29, 2026
Merged
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@HweinstockHweinstockHweinstock approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@tejaskash@agentcore-cli-automation@Hweinstock
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
feat(tests): add E2E and integration tests for Harness primitive - #1021
Merged
tejaskash merged 4 commits into
previewfrom 
feat/harness-e2e-integ-tests
Apr 29, 2026
Merged
feat(tests): add E2E and integration tests for Harness primitive#1021
tejaskash merged 4 commits into
previewfrom 
feat/harness-e2e-integ-tests
Conversation
@tejaskash
@tejaskashtejaskash
commented
Apr 29, 2026

Copy link


Copy Markdown

Contributor
Summary
  • Adds E2E tests for Harness primitive across all three model providers (Bedrock, OpenAI, Gemini), each covering create → deploy → invoke → status → teardown against real AWS
  • Adds integration tests (15 cases) covering add/remove lifecycle, config options (truncation, lifecycle, no-memory, non-Bedrock providers), validation errors, and project-level create-with-harness
  • Adds harness-bedrock to the PR baseline E2E run in CI
  • OpenAI and Gemini E2E tests skip memory (--no-harness-memory) to reduce deploy time; Bedrock keeps memory for full coverage
Design
Uses a factory pattern (createHarnessE2ESuite) mirroring the existing agent E2E helper. Each provider gets its own test file so vitest runs all three in parallel. Tests within each file are sequential since they depend on prior steps (deploy before invoke before status).
Test plan
  •  All 15 integ tests pass (npm run test:integ -- add-remove-harness)
  •  All 9 E2E tests pass against real AWS (3 providers × 3 tests each)
  •  CI workflow includes harness-bedrock in PR baseline
  •  CI passes on this PR
@tejaskash
feat(tests): add E2E and integ tests for Harness primitive
7335079
E2E tests verify create/deploy/invoke/status against real AWS for
Bedrock, OpenAI, and Gemini providers using a factory pattern that
mirrors the existing agent E2E suite. Integ tests cover the
add/remove lifecycle, configuration options (truncation, lifecycle,
model provider), validation errors, and project scaffolding.
Harness-bedrock is added to the PR E2E baseline alongside
strands-bedrock so every PR exercises the harness deploy path.
@tejaskash
skip memory for OpenAI/Gemini E2E harness tests to reduce deploy time
9a572eb
@github-actionsgithub-actionsBot
added
the 
agentcore-harness-reviewing
AgentCore Harness review in progress
label
Apr 29, 2026
@agentcore-cli-automation
agentcore-cli-automation
commented
Apr 29, 2026

Copy link


Copy Markdown

Missing CI secrets for OpenAI/Gemini harness E2E tests
The harness-openai.test.ts and harness-gemini.test.ts suites gate on env vars OPENAI_API_KEY_ARN and GEMINI_API_KEY_ARN respectively (via skipIf(!canRun) where canRun requires the env var to be truthy). These env vars are not plumbed through .github/workflows/e2e-tests.yml — only OPENAI_API_KEY and GEMINI_API_KEY (raw keys) are set today, and those are used by the existing agent tests.
Net effect:
  • When this PR lands and a future PR touches e2e-tests/harness-openai.test.ts or e2e-tests/harness-gemini.test.ts, the file will be picked up by the changed-files detection and vitest will load the suite, but every it inside will be silently skipped. CI will report green with zero assertions run — a false-positive signal.
  • The PR description claims "All 9 E2E tests pass against real AWS (3 providers × 3 tests each)", but only the 3 Bedrock tests will actually execute in CI.
Options:
  1. Provision OPENAI_API_KEY_ARN / GEMINI_API_KEY_ARN in the E2E secret bundle (Secrets Manager ARNs pointing at existing key secrets) and add them to the env: block of the Run E2E tests step. This is the "real" fix and matches the PR's stated test plan.
  2. Drop the harness-openai.test.ts / harness-gemini.test.ts files from this PR until the ARN secrets are wired up. The createHarnessE2ESuite factory can stay for later reuse.
  3. If the env-var name is the blocker, derive the ARN at test time from the existing raw key (e.g. create an ephemeral Secrets Manager secret in beforeAll and tear it down in afterAll). This keeps the CI secret surface unchanged but adds non-trivial test plumbing.
At minimum, if option 1 isn't done before merge, the tests will look like they're running but aren't — which defeats the purpose of adding them.
@github-actionsgithub-actionsBot
removed
the 
agentcore-harness-reviewing
AgentCore Harness review in progress
label
Apr 29, 2026
Hweinstock
Hweinstock
previously approved these changes
Apr 29, 2026
View reviewed changes
@HweinstockHweinstock
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice!
Comment threadinteg-tests/add-remove-harness.test.ts
Outdated

@tejaskash
fix: use shared readProjectConfig from test-utils
191f80a
Address review comment — reuse existing config-reader utility instead of
inline helper.
@tejaskash
tejaskash
requested a review
from a teamApril 29, 2026 19:02
@tejaskash
tejaskash
commented
Apr 29, 2026

Copy link


Copy Markdown

ContributorAuthor
Addressed @Hweinstock's review comment: replaced inline readProjectConfig with the shared utility from src/test-utils/config-reader.ts (191f80a).
@github-actions
github-actionsBot
commented
Apr 29, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Coverage Report
StatusCategoryPercentageCovered / Total
🔵Lines43.43%8702 / 20036
🔵Statements42.77%9213 / 21537
🔵Functions40.95%1468 / 3584
🔵Branches40.67%5678 / 13960
Generated in workflow #2139 for commit e8bc7bb by the Vitest Coverage Report Action
@tejaskash
fix: add non-null assertion to fix typecheck failure
e8bc7bb
The Zod-validated return type from readProjectConfig makes the find()
result possibly undefined. The preceding expect() guards against it at
runtime.
@tejaskash
tejaskash
merged commit 626872f
into
previewApr 29, 2026

15 checks passed

@tejaskash
tejaskash
deleted the

feat/harness-e2e-integ-tests

branch
April 29, 2026 19:34
@agentcore-cli-automationagentcore-cli-automation
mentioned this pull request
Apr 29, 2026
 Merged

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@HweinstockHweinstockHweinstock approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@tejaskash@agentcore-cli-automation@Hweinstock

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
feat(tests): add E2E and integration tests for Harness primitive - #1021
Merged
tejaskash merged 4 commits into
previewfrom 
feat/harness-e2e-integ-tests
Apr 29, 2026
Merged
feat(tests): add E2E and integration tests for Harness primitive#1021
tejaskash merged 4 commits into
previewfrom 
feat/harness-e2e-integ-tests
Conversation
@tejaskash
@tejaskashtejaskash
commented
Apr 29, 2026

Copy link


Copy Markdown

Contributor
Summary
  • Adds E2E tests for Harness primitive across all three model providers (Bedrock, OpenAI, Gemini), each covering create → deploy → invoke → status → teardown against real AWS
  • Adds integration tests (15 cases) covering add/remove lifecycle, config options (truncation, lifecycle, no-memory, non-Bedrock providers), validation errors, and project-level create-with-harness
  • Adds harness-bedrock to the PR baseline E2E run in CI
  • OpenAI and Gemini E2E tests skip memory (--no-harness-memory) to reduce deploy time; Bedrock keeps memory for full coverage
Design
Uses a factory pattern (createHarnessE2ESuite) mirroring the existing agent E2E helper. Each provider gets its own test file so vitest runs all three in parallel. Tests within each file are sequential since they depend on prior steps (deploy before invoke before status).
Test plan
  •  All 15 integ tests pass (npm run test:integ -- add-remove-harness)
  •  All 9 E2E tests pass against real AWS (3 providers × 3 tests each)
  •  CI workflow includes harness-bedrock in PR baseline
  •  CI passes on this PR
@tejaskash
feat(tests): add E2E and integ tests for Harness primitive
7335079
E2E tests verify create/deploy/invoke/status against real AWS for
Bedrock, OpenAI, and Gemini providers using a factory pattern that
mirrors the existing agent E2E suite. Integ tests cover the
add/remove lifecycle, configuration options (truncation, lifecycle,
model provider), validation errors, and project scaffolding.
Harness-bedrock is added to the PR E2E baseline alongside
strands-bedrock so every PR exercises the harness deploy path.
@tejaskash
skip memory for OpenAI/Gemini E2E harness tests to reduce deploy time
9a572eb
@github-actionsgithub-actionsBot
added
the 
agentcore-harness-reviewing
AgentCore Harness review in progress
label
Apr 29, 2026
@agentcore-cli-automation
agentcore-cli-automation
commented
Apr 29, 2026

Copy link


Copy Markdown

Missing CI secrets for OpenAI/Gemini harness E2E tests
The harness-openai.test.ts and harness-gemini.test.ts suites gate on env vars OPENAI_API_KEY_ARN and GEMINI_API_KEY_ARN respectively (via skipIf(!canRun) where canRun requires the env var to be truthy). These env vars are not plumbed through .github/workflows/e2e-tests.yml — only OPENAI_API_KEY and GEMINI_API_KEY (raw keys) are set today, and those are used by the existing agent tests.
Net effect:
  • When this PR lands and a future PR touches e2e-tests/harness-openai.test.ts or e2e-tests/harness-gemini.test.ts, the file will be picked up by the changed-files detection and vitest will load the suite, but every it inside will be silently skipped. CI will report green with zero assertions run — a false-positive signal.
  • The PR description claims "All 9 E2E tests pass against real AWS (3 providers × 3 tests each)", but only the 3 Bedrock tests will actually execute in CI.
Options:
  1. Provision OPENAI_API_KEY_ARN / GEMINI_API_KEY_ARN in the E2E secret bundle (Secrets Manager ARNs pointing at existing key secrets) and add them to the env: block of the Run E2E tests step. This is the "real" fix and matches the PR's stated test plan.
  2. Drop the harness-openai.test.ts / harness-gemini.test.ts files from this PR until the ARN secrets are wired up. The createHarnessE2ESuite factory can stay for later reuse.
  3. If the env-var name is the blocker, derive the ARN at test time from the existing raw key (e.g. create an ephemeral Secrets Manager secret in beforeAll and tear it down in afterAll). This keeps the CI secret surface unchanged but adds non-trivial test plumbing.
At minimum, if option 1 isn't done before merge, the tests will look like they're running but aren't — which defeats the purpose of adding them.
@github-actionsgithub-actionsBot
removed
the 
agentcore-harness-reviewing
AgentCore Harness review in progress
label
Apr 29, 2026
Hweinstock
Hweinstock
previously approved these changes
Apr 29, 2026
View reviewed changes
@HweinstockHweinstock
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice!
Comment threadinteg-tests/add-remove-harness.test.ts
Outdated

@tejaskash
fix: use shared readProjectConfig from test-utils
191f80a
Address review comment — reuse existing config-reader utility instead of
inline helper.
@tejaskash
tejaskash
requested a review
from a teamApril 29, 2026 19:02
@tejaskash
tejaskash
commented
Apr 29, 2026

Copy link


Copy Markdown

ContributorAuthor
Addressed @Hweinstock's review comment: replaced inline readProjectConfig with the shared utility from src/test-utils/config-reader.ts (191f80a).
@github-actions
github-actionsBot
commented
Apr 29, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Coverage Report
StatusCategoryPercentageCovered / Total
🔵Lines43.43%8702 / 20036
🔵Statements42.77%9213 / 21537
🔵Functions40.95%1468 / 3584
🔵Branches40.67%5678 / 13960
Generated in workflow #2139 for commit e8bc7bb by the Vitest Coverage Report Action
@tejaskash
fix: add non-null assertion to fix typecheck failure
e8bc7bb
The Zod-validated return type from readProjectConfig makes the find()
result possibly undefined. The preceding expect() guards against it at
runtime.
@tejaskash
tejaskash
merged commit 626872f
into
previewApr 29, 2026

15 checks passed

@tejaskash
tejaskash
deleted the

feat/harness-e2e-integ-tests

branch
April 29, 2026 19:34
@agentcore-cli-automationagentcore-cli-automation
mentioned this pull request
Apr 29, 2026
 Merged

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@HweinstockHweinstockHweinstock approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@tejaskash@agentcore-cli-automation@Hweinstock

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
feat(tests): add E2E and integration tests for Harness primitive - #1021
Merged
tejaskash merged 4 commits into
previewfrom 
feat/harness-e2e-integ-tests
Apr 29, 2026
Merged
feat(tests): add E2E and integration tests for Harness primitive#1021
tejaskash merged 4 commits into
previewfrom 
feat/harness-e2e-integ-tests
Conversation
@tejaskash
@tejaskashtejaskash
commented
Apr 29, 2026

Copy link


Copy Markdown

Contributor
Summary
  • Adds E2E tests for Harness primitive across all three model providers (Bedrock, OpenAI, Gemini), each covering create → deploy → invoke → status → teardown against real AWS
  • Adds integration tests (15 cases) covering add/remove lifecycle, config options (truncation, lifecycle, no-memory, non-Bedrock providers), validation errors, and project-level create-with-harness
  • Adds harness-bedrock to the PR baseline E2E run in CI
  • OpenAI and Gemini E2E tests skip memory (--no-harness-memory) to reduce deploy time; Bedrock keeps memory for full coverage
Design
Uses a factory pattern (createHarnessE2ESuite) mirroring the existing agent E2E helper. Each provider gets its own test file so vitest runs all three in parallel. Tests within each file are sequential since they depend on prior steps (deploy before invoke before status).
Test plan
  •  All 15 integ tests pass (npm run test:integ -- add-remove-harness)
  •  All 9 E2E tests pass against real AWS (3 providers × 3 tests each)
  •  CI workflow includes harness-bedrock in PR baseline
  •  CI passes on this PR
@tejaskash
feat(tests): add E2E and integ tests for Harness primitive
7335079
E2E tests verify create/deploy/invoke/status against real AWS for
Bedrock, OpenAI, and Gemini providers using a factory pattern that
mirrors the existing agent E2E suite. Integ tests cover the
add/remove lifecycle, configuration options (truncation, lifecycle,
model provider), validation errors, and project scaffolding.
Harness-bedrock is added to the PR E2E baseline alongside
strands-bedrock so every PR exercises the harness deploy path.
@tejaskash
skip memory for OpenAI/Gemini E2E harness tests to reduce deploy time
9a572eb
@github-actionsgithub-actionsBot
added
the 
agentcore-harness-reviewing
AgentCore Harness review in progress
label
Apr 29, 2026
@agentcore-cli-automation
agentcore-cli-automation
commented
Apr 29, 2026

Copy link


Copy Markdown

Missing CI secrets for OpenAI/Gemini harness E2E tests
The harness-openai.test.ts and harness-gemini.test.ts suites gate on env vars OPENAI_API_KEY_ARN and GEMINI_API_KEY_ARN respectively (via skipIf(!canRun) where canRun requires the env var to be truthy). These env vars are not plumbed through .github/workflows/e2e-tests.yml — only OPENAI_API_KEY and GEMINI_API_KEY (raw keys) are set today, and those are used by the existing agent tests.
Net effect:
  • When this PR lands and a future PR touches e2e-tests/harness-openai.test.ts or e2e-tests/harness-gemini.test.ts, the file will be picked up by the changed-files detection and vitest will load the suite, but every it inside will be silently skipped. CI will report green with zero assertions run — a false-positive signal.
  • The PR description claims "All 9 E2E tests pass against real AWS (3 providers × 3 tests each)", but only the 3 Bedrock tests will actually execute in CI.
Options:
  1. Provision OPENAI_API_KEY_ARN / GEMINI_API_KEY_ARN in the E2E secret bundle (Secrets Manager ARNs pointing at existing key secrets) and add them to the env: block of the Run E2E tests step. This is the "real" fix and matches the PR's stated test plan.
  2. Drop the harness-openai.test.ts / harness-gemini.test.ts files from this PR until the ARN secrets are wired up. The createHarnessE2ESuite factory can stay for later reuse.
  3. If the env-var name is the blocker, derive the ARN at test time from the existing raw key (e.g. create an ephemeral Secrets Manager secret in beforeAll and tear it down in afterAll). This keeps the CI secret surface unchanged but adds non-trivial test plumbing.
At minimum, if option 1 isn't done before merge, the tests will look like they're running but aren't — which defeats the purpose of adding them.
@github-actionsgithub-actionsBot
removed
the 
agentcore-harness-reviewing
AgentCore Harness review in progress
label
Apr 29, 2026
Hweinstock
Hweinstock
previously approved these changes
Apr 29, 2026
View reviewed changes
@HweinstockHweinstock
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice!
Comment threadinteg-tests/add-remove-harness.test.ts
Outdated

@tejaskash
fix: use shared readProjectConfig from test-utils
191f80a
Address review comment — reuse existing config-reader utility instead of
inline helper.
@tejaskash
tejaskash
requested a review
from a teamApril 29, 2026 19:02
@tejaskash
tejaskash
commented
Apr 29, 2026

Copy link


Copy Markdown

ContributorAuthor
Addressed @Hweinstock's review comment: replaced inline readProjectConfig with the shared utility from src/test-utils/config-reader.ts (191f80a).
@github-actions
github-actionsBot
commented
Apr 29, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Coverage Report
StatusCategoryPercentageCovered / Total
🔵Lines43.43%8702 / 20036
🔵Statements42.77%9213 / 21537
🔵Functions40.95%1468 / 3584
🔵Branches40.67%5678 / 13960
Generated in workflow #2139 for commit e8bc7bb by the Vitest Coverage Report Action
@tejaskash
fix: add non-null assertion to fix typecheck failure
e8bc7bb
The Zod-validated return type from readProjectConfig makes the find()
result possibly undefined. The preceding expect() guards against it at
runtime.
@tejaskash
tejaskash
merged commit 626872f
into
previewApr 29, 2026

15 checks passed

@tejaskash
tejaskash
deleted the

feat/harness-e2e-integ-tests

branch
April 29, 2026 19:34
@agentcore-cli-automationagentcore-cli-automation
mentioned this pull request
Apr 29, 2026
 Merged

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@HweinstockHweinstockHweinstock approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@tejaskash@agentcore-cli-automation@Hweinstock

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
feat(tests): add E2E and integration tests for Harness primitive - #1021
Merged
tejaskash merged 4 commits into
previewfrom 
feat/harness-e2e-integ-tests
Apr 29, 2026
Merged
feat(tests): add E2E and integration tests for Harness primitive#1021
tejaskash merged 4 commits into
previewfrom 
feat/harness-e2e-integ-tests
Conversation
@tejaskash
@tejaskashtejaskash
commented
Apr 29, 2026

Copy link


Copy Markdown

Contributor
Summary
  • Adds E2E tests for Harness primitive across all three model providers (Bedrock, OpenAI, Gemini), each covering create → deploy → invoke → status → teardown against real AWS
  • Adds integration tests (15 cases) covering add/remove lifecycle, config options (truncation, lifecycle, no-memory, non-Bedrock providers), validation errors, and project-level create-with-harness
  • Adds harness-bedrock to the PR baseline E2E run in CI
  • OpenAI and Gemini E2E tests skip memory (--no-harness-memory) to reduce deploy time; Bedrock keeps memory for full coverage
Design
Uses a factory pattern (createHarnessE2ESuite) mirroring the existing agent E2E helper. Each provider gets its own test file so vitest runs all three in parallel. Tests within each file are sequential since they depend on prior steps (deploy before invoke before status).
Test plan
  •  All 15 integ tests pass (npm run test:integ -- add-remove-harness)
  •  All 9 E2E tests pass against real AWS (3 providers × 3 tests each)
  •  CI workflow includes harness-bedrock in PR baseline
  •  CI passes on this PR
@tejaskash
feat(tests): add E2E and integ tests for Harness primitive
7335079
E2E tests verify create/deploy/invoke/status against real AWS for
Bedrock, OpenAI, and Gemini providers using a factory pattern that
mirrors the existing agent E2E suite. Integ tests cover the
add/remove lifecycle, configuration options (truncation, lifecycle,
model provider), validation errors, and project scaffolding.
Harness-bedrock is added to the PR E2E baseline alongside
strands-bedrock so every PR exercises the harness deploy path.
@tejaskash
skip memory for OpenAI/Gemini E2E harness tests to reduce deploy time
9a572eb
@github-actionsgithub-actionsBot
added
the 
agentcore-harness-reviewing
AgentCore Harness review in progress
label
Apr 29, 2026
@agentcore-cli-automation
agentcore-cli-automation
commented
Apr 29, 2026

Copy link


Copy Markdown

Missing CI secrets for OpenAI/Gemini harness E2E tests
The harness-openai.test.ts and harness-gemini.test.ts suites gate on env vars OPENAI_API_KEY_ARN and GEMINI_API_KEY_ARN respectively (via skipIf(!canRun) where canRun requires the env var to be truthy). These env vars are not plumbed through .github/workflows/e2e-tests.yml — only OPENAI_API_KEY and GEMINI_API_KEY (raw keys) are set today, and those are used by the existing agent tests.
Net effect:
  • When this PR lands and a future PR touches e2e-tests/harness-openai.test.ts or e2e-tests/harness-gemini.test.ts, the file will be picked up by the changed-files detection and vitest will load the suite, but every it inside will be silently skipped. CI will report green with zero assertions run — a false-positive signal.
  • The PR description claims "All 9 E2E tests pass against real AWS (3 providers × 3 tests each)", but only the 3 Bedrock tests will actually execute in CI.
Options:
  1. Provision OPENAI_API_KEY_ARN / GEMINI_API_KEY_ARN in the E2E secret bundle (Secrets Manager ARNs pointing at existing key secrets) and add them to the env: block of the Run E2E tests step. This is the "real" fix and matches the PR's stated test plan.
  2. Drop the harness-openai.test.ts / harness-gemini.test.ts files from this PR until the ARN secrets are wired up. The createHarnessE2ESuite factory can stay for later reuse.
  3. If the env-var name is the blocker, derive the ARN at test time from the existing raw key (e.g. create an ephemeral Secrets Manager secret in beforeAll and tear it down in afterAll). This keeps the CI secret surface unchanged but adds non-trivial test plumbing.
At minimum, if option 1 isn't done before merge, the tests will look like they're running but aren't — which defeats the purpose of adding them.
@github-actionsgithub-actionsBot
removed
the 
agentcore-harness-reviewing
AgentCore Harness review in progress
label
Apr 29, 2026
Hweinstock
Hweinstock
previously approved these changes
Apr 29, 2026
View reviewed changes
@HweinstockHweinstock
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice!
Comment threadinteg-tests/add-remove-harness.test.ts
Outdated

@tejaskash
fix: use shared readProjectConfig from test-utils
191f80a
Address review comment — reuse existing config-reader utility instead of
inline helper.
@tejaskash
tejaskash
requested a review
from a teamApril 29, 2026 19:02
@tejaskash
tejaskash
commented
Apr 29, 2026

Copy link


Copy Markdown

ContributorAuthor
Addressed @Hweinstock's review comment: replaced inline readProjectConfig with the shared utility from src/test-utils/config-reader.ts (191f80a).
@github-actions
github-actionsBot
commented
Apr 29, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Coverage Report
StatusCategoryPercentageCovered / Total
🔵Lines43.43%8702 / 20036
🔵Statements42.77%9213 / 21537
🔵Functions40.95%1468 / 3584
🔵Branches40.67%5678 / 13960
Generated in workflow #2139 for commit e8bc7bb by the Vitest Coverage Report Action
@tejaskash
fix: add non-null assertion to fix typecheck failure
e8bc7bb
The Zod-validated return type from readProjectConfig makes the find()
result possibly undefined. The preceding expect() guards against it at
runtime.
@tejaskash
tejaskash
merged commit 626872f
into
previewApr 29, 2026

15 checks passed

@tejaskash
tejaskash
deleted the

feat/harness-e2e-integ-tests

branch
April 29, 2026 19:34
@agentcore-cli-automationagentcore-cli-automation
mentioned this pull request
Apr 29, 2026
 Merged

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@HweinstockHweinstockHweinstock approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@tejaskash@agentcore-cli-automation@Hweinstock

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
feat(tests): add E2E and integration tests for Harness primitive - #1021
Merged
tejaskash merged 4 commits into
previewfrom 
feat/harness-e2e-integ-tests
Apr 29, 2026
Merged
feat(tests): add E2E and integration tests for Harness primitive#1021
tejaskash merged 4 commits into
previewfrom 
feat/harness-e2e-integ-tests
Conversation
@tejaskash
@tejaskashtejaskash
commented
Apr 29, 2026

Copy link


Copy Markdown

Contributor
Summary
  • Adds E2E tests for Harness primitive across all three model providers (Bedrock, OpenAI, Gemini), each covering create → deploy → invoke → status → teardown against real AWS
  • Adds integration tests (15 cases) covering add/remove lifecycle, config options (truncation, lifecycle, no-memory, non-Bedrock providers), validation errors, and project-level create-with-harness
  • Adds harness-bedrock to the PR baseline E2E run in CI
  • OpenAI and Gemini E2E tests skip memory (--no-harness-memory) to reduce deploy time; Bedrock keeps memory for full coverage
Design
Uses a factory pattern (createHarnessE2ESuite) mirroring the existing agent E2E helper. Each provider gets its own test file so vitest runs all three in parallel. Tests within each file are sequential since they depend on prior steps (deploy before invoke before status).
Test plan
  •  All 15 integ tests pass (npm run test:integ -- add-remove-harness)
  •  All 9 E2E tests pass against real AWS (3 providers × 3 tests each)
  •  CI workflow includes harness-bedrock in PR baseline
  •  CI passes on this PR
@tejaskash
feat(tests): add E2E and integ tests for Harness primitive
7335079
E2E tests verify create/deploy/invoke/status against real AWS for
Bedrock, OpenAI, and Gemini providers using a factory pattern that
mirrors the existing agent E2E suite. Integ tests cover the
add/remove lifecycle, configuration options (truncation, lifecycle,
model provider), validation errors, and project scaffolding.
Harness-bedrock is added to the PR E2E baseline alongside
strands-bedrock so every PR exercises the harness deploy path.
@tejaskash
skip memory for OpenAI/Gemini E2E harness tests to reduce deploy time
9a572eb
@github-actionsgithub-actionsBot
added
the 
agentcore-harness-reviewing
AgentCore Harness review in progress
label
Apr 29, 2026
@agentcore-cli-automation
agentcore-cli-automation
commented
Apr 29, 2026

Copy link


Copy Markdown

Missing CI secrets for OpenAI/Gemini harness E2E tests
The harness-openai.test.ts and harness-gemini.test.ts suites gate on env vars OPENAI_API_KEY_ARN and GEMINI_API_KEY_ARN respectively (via skipIf(!canRun) where canRun requires the env var to be truthy). These env vars are not plumbed through .github/workflows/e2e-tests.yml — only OPENAI_API_KEY and GEMINI_API_KEY (raw keys) are set today, and those are used by the existing agent tests.
Net effect:
  • When this PR lands and a future PR touches e2e-tests/harness-openai.test.ts or e2e-tests/harness-gemini.test.ts, the file will be picked up by the changed-files detection and vitest will load the suite, but every it inside will be silently skipped. CI will report green with zero assertions run — a false-positive signal.
  • The PR description claims "All 9 E2E tests pass against real AWS (3 providers × 3 tests each)", but only the 3 Bedrock tests will actually execute in CI.
Options:
  1. Provision OPENAI_API_KEY_ARN / GEMINI_API_KEY_ARN in the E2E secret bundle (Secrets Manager ARNs pointing at existing key secrets) and add them to the env: block of the Run E2E tests step. This is the "real" fix and matches the PR's stated test plan.
  2. Drop the harness-openai.test.ts / harness-gemini.test.ts files from this PR until the ARN secrets are wired up. The createHarnessE2ESuite factory can stay for later reuse.
  3. If the env-var name is the blocker, derive the ARN at test time from the existing raw key (e.g. create an ephemeral Secrets Manager secret in beforeAll and tear it down in afterAll). This keeps the CI secret surface unchanged but adds non-trivial test plumbing.
At minimum, if option 1 isn't done before merge, the tests will look like they're running but aren't — which defeats the purpose of adding them.
@github-actionsgithub-actionsBot
removed
the 
agentcore-harness-reviewing
AgentCore Harness review in progress
label
Apr 29, 2026
Hweinstock
Hweinstock
previously approved these changes
Apr 29, 2026
View reviewed changes
@HweinstockHweinstock
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice!
Comment threadinteg-tests/add-remove-harness.test.ts
Outdated

@tejaskash
fix: use shared readProjectConfig from test-utils
191f80a
Address review comment — reuse existing config-reader utility instead of
inline helper.
@tejaskash
tejaskash
requested a review
from a teamApril 29, 2026 19:02
@tejaskash
tejaskash
commented
Apr 29, 2026

Copy link


Copy Markdown

ContributorAuthor
Addressed @Hweinstock's review comment: replaced inline readProjectConfig with the shared utility from src/test-utils/config-reader.ts (191f80a).
@github-actions
github-actionsBot
commented
Apr 29, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Coverage Report
StatusCategoryPercentageCovered / Total
🔵Lines43.43%8702 / 20036
🔵Statements42.77%9213 / 21537
🔵Functions40.95%1468 / 3584
🔵Branches40.67%5678 / 13960
Generated in workflow #2139 for commit e8bc7bb by the Vitest Coverage Report Action
@tejaskash
fix: add non-null assertion to fix typecheck failure
e8bc7bb
The Zod-validated return type from readProjectConfig makes the find()
result possibly undefined. The preceding expect() guards against it at
runtime.
@tejaskash
tejaskash
merged commit 626872f
into
previewApr 29, 2026

15 checks passed

@tejaskash
tejaskash
deleted the

feat/harness-e2e-integ-tests

branch
April 29, 2026 19:34
@agentcore-cli-automationagentcore-cli-automation
mentioned this pull request
Apr 29, 2026
 Merged

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@HweinstockHweinstockHweinstock approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@tejaskash@agentcore-cli-automation@Hweinstock

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
feat(tests): add E2E and integration tests for Harness primitive - #1021
Merged
tejaskash merged 4 commits into
previewfrom 
feat/harness-e2e-integ-tests
Apr 29, 2026
Merged
feat(tests): add E2E and integration tests for Harness primitive#1021
tejaskash merged 4 commits into
previewfrom 
feat/harness-e2e-integ-tests
Conversation
@tejaskash
@tejaskashtejaskash
commented
Apr 29, 2026

Copy link


Copy Markdown

Contributor
Summary
  • Adds E2E tests for Harness primitive across all three model providers (Bedrock, OpenAI, Gemini), each covering create → deploy → invoke → status → teardown against real AWS
  • Adds integration tests (15 cases) covering add/remove lifecycle, config options (truncation, lifecycle, no-memory, non-Bedrock providers), validation errors, and project-level create-with-harness
  • Adds harness-bedrock to the PR baseline E2E run in CI
  • OpenAI and Gemini E2E tests skip memory (--no-harness-memory) to reduce deploy time; Bedrock keeps memory for full coverage
Design
Uses a factory pattern (createHarnessE2ESuite) mirroring the existing agent E2E helper. Each provider gets its own test file so vitest runs all three in parallel. Tests within each file are sequential since they depend on prior steps (deploy before invoke before status).
Test plan
  •  All 15 integ tests pass (npm run test:integ -- add-remove-harness)
  •  All 9 E2E tests pass against real AWS (3 providers × 3 tests each)
  •  CI workflow includes harness-bedrock in PR baseline
  •  CI passes on this PR
@tejaskash
feat(tests): add E2E and integ tests for Harness primitive
7335079
E2E tests verify create/deploy/invoke/status against real AWS for
Bedrock, OpenAI, and Gemini providers using a factory pattern that
mirrors the existing agent E2E suite. Integ tests cover the
add/remove lifecycle, configuration options (truncation, lifecycle,
model provider), validation errors, and project scaffolding.
Harness-bedrock is added to the PR E2E baseline alongside
strands-bedrock so every PR exercises the harness deploy path.
@tejaskash
skip memory for OpenAI/Gemini E2E harness tests to reduce deploy time
9a572eb
@github-actionsgithub-actionsBot
added
the 
agentcore-harness-reviewing
AgentCore Harness review in progress
label
Apr 29, 2026
@agentcore-cli-automation
agentcore-cli-automation
commented
Apr 29, 2026

Copy link


Copy Markdown

Missing CI secrets for OpenAI/Gemini harness E2E tests
The harness-openai.test.ts and harness-gemini.test.ts suites gate on env vars OPENAI_API_KEY_ARN and GEMINI_API_KEY_ARN respectively (via skipIf(!canRun) where canRun requires the env var to be truthy). These env vars are not plumbed through .github/workflows/e2e-tests.yml — only OPENAI_API_KEY and GEMINI_API_KEY (raw keys) are set today, and those are used by the existing agent tests.
Net effect:
  • When this PR lands and a future PR touches e2e-tests/harness-openai.test.ts or e2e-tests/harness-gemini.test.ts, the file will be picked up by the changed-files detection and vitest will load the suite, but every it inside will be silently skipped. CI will report green with zero assertions run — a false-positive signal.
  • The PR description claims "All 9 E2E tests pass against real AWS (3 providers × 3 tests each)", but only the 3 Bedrock tests will actually execute in CI.
Options:
  1. Provision OPENAI_API_KEY_ARN / GEMINI_API_KEY_ARN in the E2E secret bundle (Secrets Manager ARNs pointing at existing key secrets) and add them to the env: block of the Run E2E tests step. This is the "real" fix and matches the PR's stated test plan.
  2. Drop the harness-openai.test.ts / harness-gemini.test.ts files from this PR until the ARN secrets are wired up. The createHarnessE2ESuite factory can stay for later reuse.
  3. If the env-var name is the blocker, derive the ARN at test time from the existing raw key (e.g. create an ephemeral Secrets Manager secret in beforeAll and tear it down in afterAll). This keeps the CI secret surface unchanged but adds non-trivial test plumbing.
At minimum, if option 1 isn't done before merge, the tests will look like they're running but aren't — which defeats the purpose of adding them.
@github-actionsgithub-actionsBot
removed
the 
agentcore-harness-reviewing
AgentCore Harness review in progress
label
Apr 29, 2026
Hweinstock
Hweinstock
previously approved these changes
Apr 29, 2026
View reviewed changes
@HweinstockHweinstock
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice!
Comment threadinteg-tests/add-remove-harness.test.ts
Outdated

@tejaskash
fix: use shared readProjectConfig from test-utils
191f80a
Address review comment — reuse existing config-reader utility instead of
inline helper.
@tejaskash
tejaskash
requested a review
from a teamApril 29, 2026 19:02
@tejaskash
tejaskash
commented
Apr 29, 2026

Copy link


Copy Markdown

ContributorAuthor
Addressed @Hweinstock's review comment: replaced inline readProjectConfig with the shared utility from src/test-utils/config-reader.ts (191f80a).
@github-actions
github-actionsBot
commented
Apr 29, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Coverage Report
StatusCategoryPercentageCovered / Total
🔵Lines43.43%8702 / 20036
🔵Statements42.77%9213 / 21537
🔵Functions40.95%1468 / 3584
🔵Branches40.67%5678 / 13960
Generated in workflow #2139 for commit e8bc7bb by the Vitest Coverage Report Action
@tejaskash
fix: add non-null assertion to fix typecheck failure
e8bc7bb
The Zod-validated return type from readProjectConfig makes the find()
result possibly undefined. The preceding expect() guards against it at
runtime.
@tejaskash
tejaskash
merged commit 626872f
into
previewApr 29, 2026

15 checks passed

@tejaskash
tejaskash
deleted the

feat/harness-e2e-integ-tests

branch
April 29, 2026 19:34
@agentcore-cli-automationagentcore-cli-automation
mentioned this pull request
Apr 29, 2026
 Merged

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@HweinstockHweinstockHweinstock approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@tejaskash@agentcore-cli-automation@Hweinstock

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
feat(tests): add E2E and integration tests for Harness primitive - #1021
Merged
tejaskash merged 4 commits into
previewfrom 
feat/harness-e2e-integ-tests
Apr 29, 2026
Merged
feat(tests): add E2E and integration tests for Harness primitive#1021
tejaskash merged 4 commits into
previewfrom 
feat/harness-e2e-integ-tests
Conversation
@tejaskash
@tejaskashtejaskash
commented
Apr 29, 2026

Copy link


Copy Markdown

Contributor
Summary
  • Adds E2E tests for Harness primitive across all three model providers (Bedrock, OpenAI, Gemini), each covering create → deploy → invoke → status → teardown against real AWS
  • Adds integration tests (15 cases) covering add/remove lifecycle, config options (truncation, lifecycle, no-memory, non-Bedrock providers), validation errors, and project-level create-with-harness
  • Adds harness-bedrock to the PR baseline E2E run in CI
  • OpenAI and Gemini E2E tests skip memory (--no-harness-memory) to reduce deploy time; Bedrock keeps memory for full coverage
Design
Uses a factory pattern (createHarnessE2ESuite) mirroring the existing agent E2E helper. Each provider gets its own test file so vitest runs all three in parallel. Tests within each file are sequential since they depend on prior steps (deploy before invoke before status).
Test plan
  •  All 15 integ tests pass (npm run test:integ -- add-remove-harness)
  •  All 9 E2E tests pass against real AWS (3 providers × 3 tests each)
  •  CI workflow includes harness-bedrock in PR baseline
  •  CI passes on this PR
@tejaskash
feat(tests): add E2E and integ tests for Harness primitive
7335079
E2E tests verify create/deploy/invoke/status against real AWS for
Bedrock, OpenAI, and Gemini providers using a factory pattern that
mirrors the existing agent E2E suite. Integ tests cover the
add/remove lifecycle, configuration options (truncation, lifecycle,
model provider), validation errors, and project scaffolding.
Harness-bedrock is added to the PR E2E baseline alongside
strands-bedrock so every PR exercises the harness deploy path.
@tejaskash
skip memory for OpenAI/Gemini E2E harness tests to reduce deploy time
9a572eb
@github-actionsgithub-actionsBot
added
the 
agentcore-harness-reviewing
AgentCore Harness review in progress
label
Apr 29, 2026
@agentcore-cli-automation
agentcore-cli-automation
commented
Apr 29, 2026

Copy link


Copy Markdown

Missing CI secrets for OpenAI/Gemini harness E2E tests
The harness-openai.test.ts and harness-gemini.test.ts suites gate on env vars OPENAI_API_KEY_ARN and GEMINI_API_KEY_ARN respectively (via skipIf(!canRun) where canRun requires the env var to be truthy). These env vars are not plumbed through .github/workflows/e2e-tests.yml — only OPENAI_API_KEY and GEMINI_API_KEY (raw keys) are set today, and those are used by the existing agent tests.
Net effect:
  • When this PR lands and a future PR touches e2e-tests/harness-openai.test.ts or e2e-tests/harness-gemini.test.ts, the file will be picked up by the changed-files detection and vitest will load the suite, but every it inside will be silently skipped. CI will report green with zero assertions run — a false-positive signal.
  • The PR description claims "All 9 E2E tests pass against real AWS (3 providers × 3 tests each)", but only the 3 Bedrock tests will actually execute in CI.
Options:
  1. Provision OPENAI_API_KEY_ARN / GEMINI_API_KEY_ARN in the E2E secret bundle (Secrets Manager ARNs pointing at existing key secrets) and add them to the env: block of the Run E2E tests step. This is the "real" fix and matches the PR's stated test plan.
  2. Drop the harness-openai.test.ts / harness-gemini.test.ts files from this PR until the ARN secrets are wired up. The createHarnessE2ESuite factory can stay for later reuse.
  3. If the env-var name is the blocker, derive the ARN at test time from the existing raw key (e.g. create an ephemeral Secrets Manager secret in beforeAll and tear it down in afterAll). This keeps the CI secret surface unchanged but adds non-trivial test plumbing.
At minimum, if option 1 isn't done before merge, the tests will look like they're running but aren't — which defeats the purpose of adding them.
@github-actionsgithub-actionsBot
removed
the 
agentcore-harness-reviewing
AgentCore Harness review in progress
label
Apr 29, 2026
Hweinstock
Hweinstock
previously approved these changes
Apr 29, 2026
View reviewed changes
@HweinstockHweinstock
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice!
Comment threadinteg-tests/add-remove-harness.test.ts
Outdated

@tejaskash
fix: use shared readProjectConfig from test-utils
191f80a
Address review comment — reuse existing config-reader utility instead of
inline helper.
@tejaskash
tejaskash
requested a review
from a teamApril 29, 2026 19:02
@tejaskash
tejaskash
commented
Apr 29, 2026

Copy link


Copy Markdown

ContributorAuthor
Addressed @Hweinstock's review comment: replaced inline readProjectConfig with the shared utility from src/test-utils/config-reader.ts (191f80a).
@github-actions
github-actionsBot
commented
Apr 29, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Coverage Report
StatusCategoryPercentageCovered / Total
🔵Lines43.43%8702 / 20036
🔵Statements42.77%9213 / 21537
🔵Functions40.95%1468 / 3584
🔵Branches40.67%5678 / 13960
Generated in workflow #2139 for commit e8bc7bb by the Vitest Coverage Report Action
@tejaskash
fix: add non-null assertion to fix typecheck failure
e8bc7bb
The Zod-validated return type from readProjectConfig makes the find()
result possibly undefined. The preceding expect() guards against it at
runtime.
@tejaskash
tejaskash
merged commit 626872f
into
previewApr 29, 2026

15 checks passed

@tejaskash
tejaskash
deleted the

feat/harness-e2e-integ-tests

branch
April 29, 2026 19:34
@agentcore-cli-automationagentcore-cli-automation
mentioned this pull request
Apr 29, 2026
 Merged

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@HweinstockHweinstockHweinstock approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@tejaskash@agentcore-cli-automation@Hweinstock