Skip to content

fix(ci): add pinned deepeval and autoevals to dev group - #609

Merged
jariy17 merged 1 commit into
mainfrom
fix/ci-pin-deepeval-dev
Jul 31, 2026
Merged

fix(ci): add pinned deepeval and autoevals to dev group#609
jariy17 merged 1 commit into
mainfrom
fix/ci-pin-deepeval-dev

Conversation

@jariy17

@jariy17jariy17 commented Jul 30, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

Problem

ci.yml unit tests have been failing on main since 9b7d38e (#568), on all four Python versions:

ERROR tests/.../third_party/deepeval/test_adapter.py
ERROR tests/.../third_party/deepeval/test_error_handling.py
!!!!!! Interrupted: 2 errors during collection !!!!!!
Process completed with exit code 2

src/bedrock_agentcore/evaluation/custom_code_based_evaluators/third_party/deepeval/adapter.py:6 imports deepeval at module scope:

fromdeepeval.errorsimportMissingTestCaseParamsErrorfromdeepeval.metricsimportBaseConversationalMetric, BaseMetricfromdeepeval.test_case.conversational_test_caseimportTurn

But deepeval was not in the dev dependency group, and ci.yml installs via uv sync --dev. So the tests could not even be collected — pytest exits 2, and fail-fast cancels the whole matrix.

The autoevals sibling adapter imports nothing at module scope (only inside functions), which is why only the deepeval tests error. That asymmetry is the underlying issue.

Change

Add both to the dev group, pinned with an upper bound so a new major version cannot drift in:

"deepeval>=3.5.0,<5.0.0","autoevals>=0.3.0,<1.0.0",

Also corrects an unsatisfiable floor in the deepeval extra since ConversationalTurn was introduced in v3

Testing

  • uv lock regenerated; diff adds only the two packages, no other version moved
  • uv sync --dev (CI's exact command) then uv run pytest tests/ on Python 3.10 → the 2 collection errors are gone; 2930 passed
  • Third-party adapter suite passes at both the pinned floor (deepeval==3.5.0, autoevals==0.3.0) and current latest (deepeval==4.1.4) → 108 passed each
  • 4 unrelated tests (test_label_required_without_error_code, test_evaluation_with_empty_trajectory, 2 × TestAsyncMode) fail identically on clean main — pre-existing, out of scope here

Follow-up worth considering

deepeval is declared optional in pyproject.toml, yet importing the adapter module hard-requires it. Making the deepeval adapter lazy-import like the autoevals one would mean an optional extra can't break collection again. Happy to file separately.

Separate from #608, which fixes the integration-test workflow.

jariy17
fix(ci): add pinned deepeval and autoevals to dev group
fe8cd82
The deepeval adapter imports deepeval at module scope
(src/.../third_party/deepeval/adapter.py:6), but deepeval was absent from
the dev dependency group. ci.yml runs 'uv sync --dev', so the deepeval
unit tests could not be collected, exiting pytest with code 2 and
cancelling the whole matrix.
Both are pinned with an upper bound so major versions cannot drift in.
The deepeval extra's >=2.0.0 floor was also unsatisfiable: the adapter
imports deepeval.test_case.conversational_test_case.Turn, which does not
exist before 3.0.8, and deepeval <3.5.0 pins rich <14, conflicting with
strands-agents-evals (rich >=14). Corrected to >=3.5.0.
@jariy17
jariy17 requested a review from a teamJuly 30, 2026 22:47
@github-actionsgithub-actionsBot added the size/xs PR size: XS label Jul 30, 2026
@agentcore-devx-automationagentcore-devx-automationBot added the claude-security-reviewing Claude Code /security-review in progress label Jul 30, 2026
@agentcore-devx-automation

agentcore-devx-automationBot commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Claude Security Review: no high-confidence findings. (run)

@agentcore-devx-automationagentcore-devx-automationBot removed the claude-security-reviewing Claude Code /security-review in progress label Jul 30, 2026
@github-actionsgithub-actionsBot added size/xs PR size: XS and removed size/xs PR size: XS labels Jul 30, 2026
@jariy17jariy17 mentioned this pull request Jul 30, 2026
Merged
@jariy17
jariy17 merged commit 70165d9 into mainJul 31, 2026
36 of 42 checks passed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@tejaskashtejaskashtejaskash approved these changes

Assignees

No one assigned

Labels

size/xsPR size: XS

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@jariy17@tejaskash
, 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all 
 blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
fix(ci): add pinned deepeval and autoevals to dev group by jariy17 · Pull Request #609 · aws/bedrock-agentcore-sdk-python · GitHub
Skip to content
fix(ci): add pinned deepeval and autoevals to dev group - #609
Merged
jariy17 merged 1 commit into
mainfrom 
fix/ci-pin-deepeval-dev
Jul 31, 2026
Merged
fix(ci): add pinned deepeval and autoevals to dev group#609
jariy17 merged 1 commit into
mainfrom 
fix/ci-pin-deepeval-dev
Conversation
@jariy17
@jariy17jariy17
commented
Jul 30, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Problem
ci.yml unit tests have been failing on main since 9b7d38e (#568), on all four Python versions:
ERROR tests/.../third_party/deepeval/test_adapter.py
ERROR tests/.../third_party/deepeval/test_error_handling.py
!!!!!! Interrupted: 2 errors during collection !!!!!!
Process completed with exit code 2

src/bedrock_agentcore/evaluation/custom_code_based_evaluators/third_party/deepeval/adapter.py:6 imports deepeval at module scope:
fromdeepeval.errorsimportMissingTestCaseParamsErrorfromdeepeval.metricsimportBaseConversationalMetric, BaseMetricfromdeepeval.test_case.conversational_test_caseimportTurn
But deepeval was not in the dev dependency group, and ci.yml installs via uv sync --dev. So the tests could not even be collected — pytest exits 2, and fail-fast cancels the whole matrix.
The autoevals sibling adapter imports nothing at module scope (only inside functions), which is why only the deepeval tests error. That asymmetry is the underlying issue.
Change
Add both to the dev group, pinned with an upper bound so a new major version cannot drift in:
"deepeval>=3.5.0,<5.0.0","autoevals>=0.3.0,<1.0.0",
Also corrects an unsatisfiable floor in the deepeval extra since ConversationalTurn was introduced in v3
Testing
  • uv lock regenerated; diff adds only the two packages, no other version moved
  • uv sync --dev (CI's exact command) then uv run pytest tests/ on Python 3.10 → the 2 collection errors are gone; 2930 passed
  • Third-party adapter suite passes at both the pinned floor (deepeval==3.5.0, autoevals==0.3.0) and current latest (deepeval==4.1.4) → 108 passed each
  • 4 unrelated tests (test_label_required_without_error_code, test_evaluation_with_empty_trajectory, 2 × TestAsyncMode) fail identically on clean main — pre-existing, out of scope here
Follow-up worth considering
deepeval is declared optional in pyproject.toml, yet importing the adapter module hard-requires it. Making the deepeval adapter lazy-import like the autoevals one would mean an optional extra can't break collection again. Happy to file separately.
Separate from #608, which fixes the integration-test workflow.
jariy17
fix(ci): add pinned deepeval and autoevals to dev group
fe8cd82
The deepeval adapter imports deepeval at module scope
(src/.../third_party/deepeval/adapter.py:6), but deepeval was absent from
the dev dependency group. ci.yml runs 'uv sync --dev', so the deepeval
unit tests could not be collected, exiting pytest with code 2 and
cancelling the whole matrix.
Both are pinned with an upper bound so major versions cannot drift in.
The deepeval extra's >=2.0.0 floor was also unsatisfiable: the adapter
imports deepeval.test_case.conversational_test_case.Turn, which does not
exist before 3.0.8, and deepeval <3.5.0 pins rich <14, conflicting with
strands-agents-evals (rich >=14). Corrected to >=3.5.0.
@jariy17
jariy17
requested a review
from a teamJuly 30, 2026 22:47
@github-actionsgithub-actionsBot
added
the 
size/xs
PR size: XS
label
Jul 30, 2026
@agentcore-devx-automationagentcore-devx-automationBot
added
the 
claude-security-reviewing
Claude Code /security-review in progress
label
Jul 30, 2026
@agentcore-devx-automation
agentcore-devx-automationBot
commented
Jul 30, 2026

Copy link


Copy Markdown

Contributor
Claude Security Review: no high-confidence findings. (run)
@agentcore-devx-automationagentcore-devx-automationBot
removed
the 
claude-security-reviewing
Claude Code /security-review in progress
label
Jul 30, 2026
@github-actionsgithub-actionsBot
added

size/xs
PR size: XS
and removed

size/xs
PR size: XS
labels
Jul 30, 2026
@jariy17jariy17
mentioned this pull request
Jul 30, 2026
 Merged

@jariy17
jariy17
merged commit 70165d9
into
mainJul 31, 2026

36 of 42 checks passed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@tejaskashtejaskashtejaskash approved these changes

Assignees


No one assigned


Labels

size/xsPR size: XS

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@jariy17@tejaskash

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Force GitHub README to respect dark mode
(function() {
var style = document.createElement('style');
style.textContent = '
.markdown-body {
color-scheme: dark light;
}
.markdown-body pre { background: #161b22 !important; }
.markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }
.markdown-body table th, .markdown-body table td { border-color: #30363d !important; }
.markdown-body img { background: #0d1117; }
.markdown-body blockquote { border-left-color: #8b949e; }
.markdown-body hr { border-color: #30363d; }
';
document.head.appendChild(style);
})();
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
fix(ci): add pinned deepeval and autoevals to dev group by jariy17 · Pull Request #609 · aws/bedrock-agentcore-sdk-python · GitHub
Skip to content
fix(ci): add pinned deepeval and autoevals to dev group - #609
Merged
jariy17 merged 1 commit into
mainfrom 
fix/ci-pin-deepeval-dev
Jul 31, 2026
Merged
fix(ci): add pinned deepeval and autoevals to dev group#609
jariy17 merged 1 commit into
mainfrom 
fix/ci-pin-deepeval-dev
Conversation
@jariy17
@jariy17jariy17
commented
Jul 30, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Problem
ci.yml unit tests have been failing on main since 9b7d38e (#568), on all four Python versions:
ERROR tests/.../third_party/deepeval/test_adapter.py
ERROR tests/.../third_party/deepeval/test_error_handling.py
!!!!!! Interrupted: 2 errors during collection !!!!!!
Process completed with exit code 2

src/bedrock_agentcore/evaluation/custom_code_based_evaluators/third_party/deepeval/adapter.py:6 imports deepeval at module scope:
fromdeepeval.errorsimportMissingTestCaseParamsErrorfromdeepeval.metricsimportBaseConversationalMetric, BaseMetricfromdeepeval.test_case.conversational_test_caseimportTurn
But deepeval was not in the dev dependency group, and ci.yml installs via uv sync --dev. So the tests could not even be collected — pytest exits 2, and fail-fast cancels the whole matrix.
The autoevals sibling adapter imports nothing at module scope (only inside functions), which is why only the deepeval tests error. That asymmetry is the underlying issue.
Change
Add both to the dev group, pinned with an upper bound so a new major version cannot drift in:
"deepeval>=3.5.0,<5.0.0","autoevals>=0.3.0,<1.0.0",
Also corrects an unsatisfiable floor in the deepeval extra since ConversationalTurn was introduced in v3
Testing
  • uv lock regenerated; diff adds only the two packages, no other version moved
  • uv sync --dev (CI's exact command) then uv run pytest tests/ on Python 3.10 → the 2 collection errors are gone; 2930 passed
  • Third-party adapter suite passes at both the pinned floor (deepeval==3.5.0, autoevals==0.3.0) and current latest (deepeval==4.1.4) → 108 passed each
  • 4 unrelated tests (test_label_required_without_error_code, test_evaluation_with_empty_trajectory, 2 × TestAsyncMode) fail identically on clean main — pre-existing, out of scope here
Follow-up worth considering
deepeval is declared optional in pyproject.toml, yet importing the adapter module hard-requires it. Making the deepeval adapter lazy-import like the autoevals one would mean an optional extra can't break collection again. Happy to file separately.
Separate from #608, which fixes the integration-test workflow.
jariy17
fix(ci): add pinned deepeval and autoevals to dev group
fe8cd82
The deepeval adapter imports deepeval at module scope
(src/.../third_party/deepeval/adapter.py:6), but deepeval was absent from
the dev dependency group. ci.yml runs 'uv sync --dev', so the deepeval
unit tests could not be collected, exiting pytest with code 2 and
cancelling the whole matrix.
Both are pinned with an upper bound so major versions cannot drift in.
The deepeval extra's >=2.0.0 floor was also unsatisfiable: the adapter
imports deepeval.test_case.conversational_test_case.Turn, which does not
exist before 3.0.8, and deepeval <3.5.0 pins rich <14, conflicting with
strands-agents-evals (rich >=14). Corrected to >=3.5.0.
@jariy17
jariy17
requested a review
from a teamJuly 30, 2026 22:47
@github-actionsgithub-actionsBot
added
the 
size/xs
PR size: XS
label
Jul 30, 2026
@agentcore-devx-automationagentcore-devx-automationBot
added
the 
claude-security-reviewing
Claude Code /security-review in progress
label
Jul 30, 2026
@agentcore-devx-automation
agentcore-devx-automationBot
commented
Jul 30, 2026

Copy link


Copy Markdown

Contributor
Claude Security Review: no high-confidence findings. (run)
@agentcore-devx-automationagentcore-devx-automationBot
removed
the 
claude-security-reviewing
Claude Code /security-review in progress
label
Jul 30, 2026
@github-actionsgithub-actionsBot
added

size/xs
PR size: XS
and removed

size/xs
PR size: XS
labels
Jul 30, 2026
@jariy17jariy17
mentioned this pull request
Jul 30, 2026
 Merged

@jariy17
jariy17
merged commit 70165d9
into
mainJul 31, 2026

36 of 42 checks passed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@tejaskashtejaskashtejaskash approved these changes

Assignees


No one assigned


Labels

size/xsPR size: XS

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@jariy17@tejaskash

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Highlight search terms from Google/DuckDuckGo/Bing referrer
(function() {
var ref = document.referrer;
var terms = [];
if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {
var url = new URL(ref);
var q = url.searchParams.get('q') || url.searchParams.get('p');
if (q) {
terms = q.split(/\s+/).filter(function(t) { return t.length > 2; });
}
}
if (terms.length === 0) return;
var style = document.createElement('style');
style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';
document.head.appendChild(style);
function highlight(node) {
if (node.nodeType === 3) { // text node
var text = node.textContent;
var found = false;
terms.forEach(function(term) {
var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi');
if (regex.test(text)) {
found = true;
var frag = document.createDocumentFragment();
var parts = text.split(regex);
parts.forEach(function(part, i) {
if (i % 2 === 0) {
frag.appendChild(document.createTextNode(part));
} else {
var span = document.createElement('span');
span.className = 'userscript-highlight';
span.textContent = part;
frag.appendChild(span);
}
});
node.parentNode.replaceChild(frag, node);
}
});
} else if (node.nodeType === 1 && node.childNodes) { // element
var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];
if (!skipTags.includes(node.tagName)) {
Array.from(node.childNodes).forEach(highlight);
}
}
}
highlight(document.body);
// Re-highlight on dynamic content
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1 || node.nodeType === 3) highlight(node);
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
fix(ci): add pinned deepeval and autoevals to dev group by jariy17 · Pull Request #609 · aws/bedrock-agentcore-sdk-python · GitHub
Skip to content
fix(ci): add pinned deepeval and autoevals to dev group - #609
Merged
jariy17 merged 1 commit into
mainfrom 
fix/ci-pin-deepeval-dev
Jul 31, 2026
Merged
fix(ci): add pinned deepeval and autoevals to dev group#609
jariy17 merged 1 commit into
mainfrom 
fix/ci-pin-deepeval-dev
Conversation
@jariy17
@jariy17jariy17
commented
Jul 30, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Problem
ci.yml unit tests have been failing on main since 9b7d38e (#568), on all four Python versions:
ERROR tests/.../third_party/deepeval/test_adapter.py
ERROR tests/.../third_party/deepeval/test_error_handling.py
!!!!!! Interrupted: 2 errors during collection !!!!!!
Process completed with exit code 2

src/bedrock_agentcore/evaluation/custom_code_based_evaluators/third_party/deepeval/adapter.py:6 imports deepeval at module scope:
fromdeepeval.errorsimportMissingTestCaseParamsErrorfromdeepeval.metricsimportBaseConversationalMetric, BaseMetricfromdeepeval.test_case.conversational_test_caseimportTurn
But deepeval was not in the dev dependency group, and ci.yml installs via uv sync --dev. So the tests could not even be collected — pytest exits 2, and fail-fast cancels the whole matrix.
The autoevals sibling adapter imports nothing at module scope (only inside functions), which is why only the deepeval tests error. That asymmetry is the underlying issue.
Change
Add both to the dev group, pinned with an upper bound so a new major version cannot drift in:
"deepeval>=3.5.0,<5.0.0","autoevals>=0.3.0,<1.0.0",
Also corrects an unsatisfiable floor in the deepeval extra since ConversationalTurn was introduced in v3
Testing
  • uv lock regenerated; diff adds only the two packages, no other version moved
  • uv sync --dev (CI's exact command) then uv run pytest tests/ on Python 3.10 → the 2 collection errors are gone; 2930 passed
  • Third-party adapter suite passes at both the pinned floor (deepeval==3.5.0, autoevals==0.3.0) and current latest (deepeval==4.1.4) → 108 passed each
  • 4 unrelated tests (test_label_required_without_error_code, test_evaluation_with_empty_trajectory, 2 × TestAsyncMode) fail identically on clean main — pre-existing, out of scope here
Follow-up worth considering
deepeval is declared optional in pyproject.toml, yet importing the adapter module hard-requires it. Making the deepeval adapter lazy-import like the autoevals one would mean an optional extra can't break collection again. Happy to file separately.
Separate from #608, which fixes the integration-test workflow.
jariy17
fix(ci): add pinned deepeval and autoevals to dev group
fe8cd82
The deepeval adapter imports deepeval at module scope
(src/.../third_party/deepeval/adapter.py:6), but deepeval was absent from
the dev dependency group. ci.yml runs 'uv sync --dev', so the deepeval
unit tests could not be collected, exiting pytest with code 2 and
cancelling the whole matrix.
Both are pinned with an upper bound so major versions cannot drift in.
The deepeval extra's >=2.0.0 floor was also unsatisfiable: the adapter
imports deepeval.test_case.conversational_test_case.Turn, which does not
exist before 3.0.8, and deepeval <3.5.0 pins rich <14, conflicting with
strands-agents-evals (rich >=14). Corrected to >=3.5.0.
@jariy17
jariy17
requested a review
from a teamJuly 30, 2026 22:47
@github-actionsgithub-actionsBot
added
the 
size/xs
PR size: XS
label
Jul 30, 2026
@agentcore-devx-automationagentcore-devx-automationBot
added
the 
claude-security-reviewing
Claude Code /security-review in progress
label
Jul 30, 2026
@agentcore-devx-automation
agentcore-devx-automationBot
commented
Jul 30, 2026

Copy link


Copy Markdown

Contributor
Claude Security Review: no high-confidence findings. (run)
@agentcore-devx-automationagentcore-devx-automationBot
removed
the 
claude-security-reviewing
Claude Code /security-review in progress
label
Jul 30, 2026
@github-actionsgithub-actionsBot
added

size/xs
PR size: XS
and removed

size/xs
PR size: XS
labels
Jul 30, 2026
@jariy17jariy17
mentioned this pull request
Jul 30, 2026
 Merged

@jariy17
jariy17
merged commit 70165d9
into
mainJul 31, 2026

36 of 42 checks passed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@tejaskashtejaskashtejaskash approved these changes

Assignees


No one assigned


Labels

size/xsPR size: XS

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@jariy17@tejaskash

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Strip utm_, fbclid, gclid, etc. from all links on page
(function() {
var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',
'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',
'ref', 'ref_src', 'source', 'medium', 'campaign'];
function cleanUrl(url) {
try {
var u = new URL(url, window.location.origin);
var changed = false;
trackingParams.forEach(function(p) {
if (u.searchParams.has(p)) {
u.searchParams.delete(p);
changed = true;
}
});
return changed ? u.toString() : url;
} catch (e) {
return url;
}
}
function cleanLinks() {
document.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
cleanLinks();
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1) {
if (node.tagName === 'A') cleanLinks();
node.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '
fix(ci): add pinned deepeval and autoevals to dev group by jariy17 · Pull Request #609 · aws/bedrock-agentcore-sdk-python · GitHub
Skip to content
fix(ci): add pinned deepeval and autoevals to dev group - #609
Merged
jariy17 merged 1 commit into
mainfrom 
fix/ci-pin-deepeval-dev
Jul 31, 2026
Merged
fix(ci): add pinned deepeval and autoevals to dev group#609
jariy17 merged 1 commit into
mainfrom 
fix/ci-pin-deepeval-dev
Conversation
@jariy17
@jariy17jariy17
commented
Jul 30, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Problem
ci.yml unit tests have been failing on main since 9b7d38e (#568), on all four Python versions:
ERROR tests/.../third_party/deepeval/test_adapter.py
ERROR tests/.../third_party/deepeval/test_error_handling.py
!!!!!! Interrupted: 2 errors during collection !!!!!!
Process completed with exit code 2

src/bedrock_agentcore/evaluation/custom_code_based_evaluators/third_party/deepeval/adapter.py:6 imports deepeval at module scope:
fromdeepeval.errorsimportMissingTestCaseParamsErrorfromdeepeval.metricsimportBaseConversationalMetric, BaseMetricfromdeepeval.test_case.conversational_test_caseimportTurn
But deepeval was not in the dev dependency group, and ci.yml installs via uv sync --dev. So the tests could not even be collected — pytest exits 2, and fail-fast cancels the whole matrix.
The autoevals sibling adapter imports nothing at module scope (only inside functions), which is why only the deepeval tests error. That asymmetry is the underlying issue.
Change
Add both to the dev group, pinned with an upper bound so a new major version cannot drift in:
"deepeval>=3.5.0,<5.0.0","autoevals>=0.3.0,<1.0.0",
Also corrects an unsatisfiable floor in the deepeval extra since ConversationalTurn was introduced in v3
Testing
  • uv lock regenerated; diff adds only the two packages, no other version moved
  • uv sync --dev (CI's exact command) then uv run pytest tests/ on Python 3.10 → the 2 collection errors are gone; 2930 passed
  • Third-party adapter suite passes at both the pinned floor (deepeval==3.5.0, autoevals==0.3.0) and current latest (deepeval==4.1.4) → 108 passed each
  • 4 unrelated tests (test_label_required_without_error_code, test_evaluation_with_empty_trajectory, 2 × TestAsyncMode) fail identically on clean main — pre-existing, out of scope here
Follow-up worth considering
deepeval is declared optional in pyproject.toml, yet importing the adapter module hard-requires it. Making the deepeval adapter lazy-import like the autoevals one would mean an optional extra can't break collection again. Happy to file separately.
Separate from #608, which fixes the integration-test workflow.
jariy17
fix(ci): add pinned deepeval and autoevals to dev group
fe8cd82
The deepeval adapter imports deepeval at module scope
(src/.../third_party/deepeval/adapter.py:6), but deepeval was absent from
the dev dependency group. ci.yml runs 'uv sync --dev', so the deepeval
unit tests could not be collected, exiting pytest with code 2 and
cancelling the whole matrix.
Both are pinned with an upper bound so major versions cannot drift in.
The deepeval extra's >=2.0.0 floor was also unsatisfiable: the adapter
imports deepeval.test_case.conversational_test_case.Turn, which does not
exist before 3.0.8, and deepeval <3.5.0 pins rich <14, conflicting with
strands-agents-evals (rich >=14). Corrected to >=3.5.0.
@jariy17
jariy17
requested a review
from a teamJuly 30, 2026 22:47
@github-actionsgithub-actionsBot
added
the 
size/xs
PR size: XS
label
Jul 30, 2026
@agentcore-devx-automationagentcore-devx-automationBot
added
the 
claude-security-reviewing
Claude Code /security-review in progress
label
Jul 30, 2026
@agentcore-devx-automation
agentcore-devx-automationBot
commented
Jul 30, 2026

Copy link


Copy Markdown

Contributor
Claude Security Review: no high-confidence findings. (run)
@agentcore-devx-automationagentcore-devx-automationBot
removed
the 
claude-security-reviewing
Claude Code /security-review in progress
label
Jul 30, 2026
@github-actionsgithub-actionsBot
added

size/xs
PR size: XS
and removed

size/xs
PR size: XS
labels
Jul 30, 2026
@jariy17jariy17
mentioned this pull request
Jul 30, 2026
 Merged

@jariy17
jariy17
merged commit 70165d9
into
mainJul 31, 2026

36 of 42 checks passed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@tejaskashtejaskashtejaskash approved these changes

Assignees


No one assigned


Labels

size/xsPR size: XS

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@jariy17@tejaskash

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Auto-enable theater mode on YouTube
(function() {
function tryTheater() {
var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]');
if (btn && !btn.classList.contains('activated')) {
btn.click();
}
}
// Try immediately
tryTheater();
// Try after navigation (SPA)
var lastUrl = location.href;
setInterval(function() {
if (location.href !== lastUrl) {
lastUrl = location.href;
setTimeout(tryTheater, 500);
}
}, 1000);
// Also try on player load
var observer = new MutationObserver(tryTheater);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
fix(ci): add pinned deepeval and autoevals to dev group by jariy17 · Pull Request #609 · aws/bedrock-agentcore-sdk-python · GitHub
Skip to content
fix(ci): add pinned deepeval and autoevals to dev group - #609
Merged
jariy17 merged 1 commit into
mainfrom 
fix/ci-pin-deepeval-dev
Jul 31, 2026
Merged
fix(ci): add pinned deepeval and autoevals to dev group#609
jariy17 merged 1 commit into
mainfrom 
fix/ci-pin-deepeval-dev
Conversation
@jariy17
@jariy17jariy17
commented
Jul 30, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Problem
ci.yml unit tests have been failing on main since 9b7d38e (#568), on all four Python versions:
ERROR tests/.../third_party/deepeval/test_adapter.py
ERROR tests/.../third_party/deepeval/test_error_handling.py
!!!!!! Interrupted: 2 errors during collection !!!!!!
Process completed with exit code 2

src/bedrock_agentcore/evaluation/custom_code_based_evaluators/third_party/deepeval/adapter.py:6 imports deepeval at module scope:
fromdeepeval.errorsimportMissingTestCaseParamsErrorfromdeepeval.metricsimportBaseConversationalMetric, BaseMetricfromdeepeval.test_case.conversational_test_caseimportTurn
But deepeval was not in the dev dependency group, and ci.yml installs via uv sync --dev. So the tests could not even be collected — pytest exits 2, and fail-fast cancels the whole matrix.
The autoevals sibling adapter imports nothing at module scope (only inside functions), which is why only the deepeval tests error. That asymmetry is the underlying issue.
Change
Add both to the dev group, pinned with an upper bound so a new major version cannot drift in:
"deepeval>=3.5.0,<5.0.0","autoevals>=0.3.0,<1.0.0",
Also corrects an unsatisfiable floor in the deepeval extra since ConversationalTurn was introduced in v3
Testing
  • uv lock regenerated; diff adds only the two packages, no other version moved
  • uv sync --dev (CI's exact command) then uv run pytest tests/ on Python 3.10 → the 2 collection errors are gone; 2930 passed
  • Third-party adapter suite passes at both the pinned floor (deepeval==3.5.0, autoevals==0.3.0) and current latest (deepeval==4.1.4) → 108 passed each
  • 4 unrelated tests (test_label_required_without_error_code, test_evaluation_with_empty_trajectory, 2 × TestAsyncMode) fail identically on clean main — pre-existing, out of scope here
Follow-up worth considering
deepeval is declared optional in pyproject.toml, yet importing the adapter module hard-requires it. Making the deepeval adapter lazy-import like the autoevals one would mean an optional extra can't break collection again. Happy to file separately.
Separate from #608, which fixes the integration-test workflow.
jariy17
fix(ci): add pinned deepeval and autoevals to dev group
fe8cd82
The deepeval adapter imports deepeval at module scope
(src/.../third_party/deepeval/adapter.py:6), but deepeval was absent from
the dev dependency group. ci.yml runs 'uv sync --dev', so the deepeval
unit tests could not be collected, exiting pytest with code 2 and
cancelling the whole matrix.
Both are pinned with an upper bound so major versions cannot drift in.
The deepeval extra's >=2.0.0 floor was also unsatisfiable: the adapter
imports deepeval.test_case.conversational_test_case.Turn, which does not
exist before 3.0.8, and deepeval <3.5.0 pins rich <14, conflicting with
strands-agents-evals (rich >=14). Corrected to >=3.5.0.
@jariy17
jariy17
requested a review
from a teamJuly 30, 2026 22:47
@github-actionsgithub-actionsBot
added
the 
size/xs
PR size: XS
label
Jul 30, 2026
@agentcore-devx-automationagentcore-devx-automationBot
added
the 
claude-security-reviewing
Claude Code /security-review in progress
label
Jul 30, 2026
@agentcore-devx-automation
agentcore-devx-automationBot
commented
Jul 30, 2026

Copy link


Copy Markdown

Contributor
Claude Security Review: no high-confidence findings. (run)
@agentcore-devx-automationagentcore-devx-automationBot
removed
the 
claude-security-reviewing
Claude Code /security-review in progress
label
Jul 30, 2026
@github-actionsgithub-actionsBot
added

size/xs
PR size: XS
and removed

size/xs
PR size: XS
labels
Jul 30, 2026
@jariy17jariy17
mentioned this pull request
Jul 30, 2026
 Merged

@jariy17
jariy17
merged commit 70165d9
into
mainJul 31, 2026

36 of 42 checks passed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@tejaskashtejaskashtejaskash approved these changes

Assignees


No one assigned


Labels

size/xsPR size: XS

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@jariy17@tejaskash

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Remove or un-stick sticky/fixed headers that block content
(function() {
function unstick() {
document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) {
if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {
el.style.position = 'static';
el.style.top = 'auto';
el.style.zIndex = 'auto';
}
});
}
unstick();
var observer = new MutationObserver(unstick);
observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });
})();
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
fix(ci): add pinned deepeval and autoevals to dev group by jariy17 · Pull Request #609 · aws/bedrock-agentcore-sdk-python · GitHub
Skip to content
fix(ci): add pinned deepeval and autoevals to dev group - #609
Merged
jariy17 merged 1 commit into
mainfrom 
fix/ci-pin-deepeval-dev
Jul 31, 2026
Merged
fix(ci): add pinned deepeval and autoevals to dev group#609
jariy17 merged 1 commit into
mainfrom 
fix/ci-pin-deepeval-dev
Conversation
@jariy17
@jariy17jariy17
commented
Jul 30, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Problem
ci.yml unit tests have been failing on main since 9b7d38e (#568), on all four Python versions:
ERROR tests/.../third_party/deepeval/test_adapter.py
ERROR tests/.../third_party/deepeval/test_error_handling.py
!!!!!! Interrupted: 2 errors during collection !!!!!!
Process completed with exit code 2

src/bedrock_agentcore/evaluation/custom_code_based_evaluators/third_party/deepeval/adapter.py:6 imports deepeval at module scope:
fromdeepeval.errorsimportMissingTestCaseParamsErrorfromdeepeval.metricsimportBaseConversationalMetric, BaseMetricfromdeepeval.test_case.conversational_test_caseimportTurn
But deepeval was not in the dev dependency group, and ci.yml installs via uv sync --dev. So the tests could not even be collected — pytest exits 2, and fail-fast cancels the whole matrix.
The autoevals sibling adapter imports nothing at module scope (only inside functions), which is why only the deepeval tests error. That asymmetry is the underlying issue.
Change
Add both to the dev group, pinned with an upper bound so a new major version cannot drift in:
"deepeval>=3.5.0,<5.0.0","autoevals>=0.3.0,<1.0.0",
Also corrects an unsatisfiable floor in the deepeval extra since ConversationalTurn was introduced in v3
Testing
  • uv lock regenerated; diff adds only the two packages, no other version moved
  • uv sync --dev (CI's exact command) then uv run pytest tests/ on Python 3.10 → the 2 collection errors are gone; 2930 passed
  • Third-party adapter suite passes at both the pinned floor (deepeval==3.5.0, autoevals==0.3.0) and current latest (deepeval==4.1.4) → 108 passed each
  • 4 unrelated tests (test_label_required_without_error_code, test_evaluation_with_empty_trajectory, 2 × TestAsyncMode) fail identically on clean main — pre-existing, out of scope here
Follow-up worth considering
deepeval is declared optional in pyproject.toml, yet importing the adapter module hard-requires it. Making the deepeval adapter lazy-import like the autoevals one would mean an optional extra can't break collection again. Happy to file separately.
Separate from #608, which fixes the integration-test workflow.
jariy17
fix(ci): add pinned deepeval and autoevals to dev group
fe8cd82
The deepeval adapter imports deepeval at module scope
(src/.../third_party/deepeval/adapter.py:6), but deepeval was absent from
the dev dependency group. ci.yml runs 'uv sync --dev', so the deepeval
unit tests could not be collected, exiting pytest with code 2 and
cancelling the whole matrix.
Both are pinned with an upper bound so major versions cannot drift in.
The deepeval extra's >=2.0.0 floor was also unsatisfiable: the adapter
imports deepeval.test_case.conversational_test_case.Turn, which does not
exist before 3.0.8, and deepeval <3.5.0 pins rich <14, conflicting with
strands-agents-evals (rich >=14). Corrected to >=3.5.0.
@jariy17
jariy17
requested a review
from a teamJuly 30, 2026 22:47
@github-actionsgithub-actionsBot
added
the 
size/xs
PR size: XS
label
Jul 30, 2026
@agentcore-devx-automationagentcore-devx-automationBot
added
the 
claude-security-reviewing
Claude Code /security-review in progress
label
Jul 30, 2026
@agentcore-devx-automation
agentcore-devx-automationBot
commented
Jul 30, 2026

Copy link


Copy Markdown

Contributor
Claude Security Review: no high-confidence findings. (run)
@agentcore-devx-automationagentcore-devx-automationBot
removed
the 
claude-security-reviewing
Claude Code /security-review in progress
label
Jul 30, 2026
@github-actionsgithub-actionsBot
added

size/xs
PR size: XS
and removed

size/xs
PR size: XS
labels
Jul 30, 2026
@jariy17jariy17
mentioned this pull request
Jul 30, 2026
 Merged

@jariy17
jariy17
merged commit 70165d9
into
mainJul 31, 2026

36 of 42 checks passed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@tejaskashtejaskashtejaskash approved these changes

Assignees


No one assigned


Labels

size/xsPR size: XS

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@jariy17@tejaskash

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Universal Dark Mode - works on any site
(function() {
var enabled = true;
function applyDarkMode() {
if (!enabled) return;
// Create style element if it doesn't exist
var style = document.getElementById('universal-dark-mode-style');
if (!style) {
style = document.createElement('style');
style.id = 'universal-dark-mode-style';
document.head.appendChild(style);
}
// Dark mode CSS - inverts colors but preserves images/video
style.textContent = '
/* Invert everything except media */
html {
filter: invert(1) hue-rotate(180deg) !important;
background: #1a1a2e !important;
}
/* Restore images, videos, iframes, canvas */
img, video, iframe, canvas, svg, picture, [style*="background-image"] {
filter: invert(1) hue-rotate(180deg) !important;
}
/* Preserve specific elements that should not be inverted */
.no-dark-mode, .no-dark-mode *,
[data-theme="light"], [data-theme="light"],
.ace_editor, .ace_editor *,
.CodeMirror, .CodeMirror *,
.monaco-editor, .monaco-editor *,
.markdown-body pre, .markdown-body pre *,
.highlight, .highlight *,
pre code, pre code * {
filter: none !important;
}
/* Fix common UI elements */
.modal, .popup, .dropdown-menu, .tooltip, .popover {
filter: invert(1) hue-rotate(180deg) !important;
background: #2d2d44 !important;
border-color: #444 !important;
}
/* Scrollbars */
::-webkit-scrollbar { background: #1a1a2e !important; }
::-webkit-scrollbar-thumb { background: #444 !important; }
::-webkit-scrollbar-thumb:hover { background: #555 !important; }
/* Selection */
::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
';
}
function removeDarkMode() {
var style = document.getElementById('universal-dark-mode-style');
if (style) style.remove();
}
// Toggle with Alt+Shift+D
document.addEventListener('keydown', function(e) {
if (e.altKey && e.shiftKey && e.key === 'D') {
e.preventDefault();
enabled = !enabled;
if (enabled) {
applyDarkMode();
console.log('[Universal Dark Mode] Enabled');
} else {
removeDarkMode();
console.log('[Universal Dark Mode] Disabled');
}
}
});
// Apply on load
applyDarkMode();
// Re-apply on dynamic content
var observer = new MutationObserver(function(mutations) {
if (enabled && !document.getElementById('universal-dark-mode-style')) {
applyDarkMode();
}
});
observer.observe(document.head, { childList: true });
console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');
})();
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();
fix(ci): add pinned deepeval and autoevals to dev group by jariy17 · Pull Request #609 · aws/bedrock-agentcore-sdk-python · GitHub
Skip to content
fix(ci): add pinned deepeval and autoevals to dev group - #609
Merged
jariy17 merged 1 commit into
mainfrom 
fix/ci-pin-deepeval-dev
Jul 31, 2026
Merged
fix(ci): add pinned deepeval and autoevals to dev group#609
jariy17 merged 1 commit into
mainfrom 
fix/ci-pin-deepeval-dev
Conversation
@jariy17
@jariy17jariy17
commented
Jul 30, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Problem
ci.yml unit tests have been failing on main since 9b7d38e (#568), on all four Python versions:
ERROR tests/.../third_party/deepeval/test_adapter.py
ERROR tests/.../third_party/deepeval/test_error_handling.py
!!!!!! Interrupted: 2 errors during collection !!!!!!
Process completed with exit code 2

src/bedrock_agentcore/evaluation/custom_code_based_evaluators/third_party/deepeval/adapter.py:6 imports deepeval at module scope:
fromdeepeval.errorsimportMissingTestCaseParamsErrorfromdeepeval.metricsimportBaseConversationalMetric, BaseMetricfromdeepeval.test_case.conversational_test_caseimportTurn
But deepeval was not in the dev dependency group, and ci.yml installs via uv sync --dev. So the tests could not even be collected — pytest exits 2, and fail-fast cancels the whole matrix.
The autoevals sibling adapter imports nothing at module scope (only inside functions), which is why only the deepeval tests error. That asymmetry is the underlying issue.
Change
Add both to the dev group, pinned with an upper bound so a new major version cannot drift in:
"deepeval>=3.5.0,<5.0.0","autoevals>=0.3.0,<1.0.0",
Also corrects an unsatisfiable floor in the deepeval extra since ConversationalTurn was introduced in v3
Testing
  • uv lock regenerated; diff adds only the two packages, no other version moved
  • uv sync --dev (CI's exact command) then uv run pytest tests/ on Python 3.10 → the 2 collection errors are gone; 2930 passed
  • Third-party adapter suite passes at both the pinned floor (deepeval==3.5.0, autoevals==0.3.0) and current latest (deepeval==4.1.4) → 108 passed each
  • 4 unrelated tests (test_label_required_without_error_code, test_evaluation_with_empty_trajectory, 2 × TestAsyncMode) fail identically on clean main — pre-existing, out of scope here
Follow-up worth considering
deepeval is declared optional in pyproject.toml, yet importing the adapter module hard-requires it. Making the deepeval adapter lazy-import like the autoevals one would mean an optional extra can't break collection again. Happy to file separately.
Separate from #608, which fixes the integration-test workflow.
jariy17
fix(ci): add pinned deepeval and autoevals to dev group
fe8cd82
The deepeval adapter imports deepeval at module scope
(src/.../third_party/deepeval/adapter.py:6), but deepeval was absent from
the dev dependency group. ci.yml runs 'uv sync --dev', so the deepeval
unit tests could not be collected, exiting pytest with code 2 and
cancelling the whole matrix.
Both are pinned with an upper bound so major versions cannot drift in.
The deepeval extra's >=2.0.0 floor was also unsatisfiable: the adapter
imports deepeval.test_case.conversational_test_case.Turn, which does not
exist before 3.0.8, and deepeval <3.5.0 pins rich <14, conflicting with
strands-agents-evals (rich >=14). Corrected to >=3.5.0.
@jariy17
jariy17
requested a review
from a teamJuly 30, 2026 22:47
@github-actionsgithub-actionsBot
added
the 
size/xs
PR size: XS
label
Jul 30, 2026
@agentcore-devx-automationagentcore-devx-automationBot
added
the 
claude-security-reviewing
Claude Code /security-review in progress
label
Jul 30, 2026
@agentcore-devx-automation
agentcore-devx-automationBot
commented
Jul 30, 2026

Copy link


Copy Markdown

Contributor
Claude Security Review: no high-confidence findings. (run)
@agentcore-devx-automationagentcore-devx-automationBot
removed
the 
claude-security-reviewing
Claude Code /security-review in progress
label
Jul 30, 2026
@github-actionsgithub-actionsBot
added

size/xs
PR size: XS
and removed

size/xs
PR size: XS
labels
Jul 30, 2026
@jariy17jariy17
mentioned this pull request
Jul 30, 2026
 Merged

@jariy17
jariy17
merged commit 70165d9
into
mainJul 31, 2026

36 of 42 checks passed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@tejaskashtejaskashtejaskash approved these changes

Assignees


No one assigned


Labels

size/xsPR size: XS

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@jariy17@tejaskash