fix(test): pin autoevals judge to the OpenAI API

[jariy17]

Summary

Follow-up to #614, which took the evaluation group from 6 failures to 3. The three DeepEval tests now pass — confirming both the span-fixture fix and the OPENAI_API_KEY wiring work. The three remaining AutoEvalsAdapter tests fail with:

openai.AuthenticationError: Braintrust gateway error: auth failed
[401 Unauthorized]: Invalid API Key ... org: None

The key is valid — it was being sent to the wrong service.autoevals does not call OpenAI directly by default (autoevals/oai.py):

ifbase_urlisNone:
base_url=os.environ.get("OPENAI_BASE_URL") or_gateway_url() # https://gateway.braintrust.devif_is_gateway_url(base_url):
api_key=os.environ.get("BRAINTRUST_API_KEY") oros.environ.get("OPENAI_API_KEY")

With OPENAI_BASE_URL unset it routes to the Braintrust AI gateway, which rejects an OpenAI key. This is documented behaviour, not a bug — init()'s own docstring example pairs the gateway base_url with BRAINTRUST_API_KEY.

Fix

Pass an explicit client, which is the provider-selection path autoevals documents. Its api_key / base_url arguments are marked "Deprecated: Use the client argument", so this is the forward-compatible option rather than setting OPENAI_BASE_URL in CI.

The client is built once in a fixture; each test passes it to the scorer, keeping the provider choice visible at the call site.

Scoped to the Autoevals tests only — DeepEval is untouched and already passing.

Testing

# an injected client bypasses the gateway: probe with a deliberately invalid key
Error code: 401 - Incorrect API key provided: sk-fake-*robe ...
# ^ OpenAI's own error, no Braintrust in the message (previously: "Braintrust gateway error")
ruff format --check -> 1 file already formatted
ruff check -> All checks passed!

I could not run these tests against the real key locally: the value lives only in the central DevX Secrets Manager account, and secretsmanager:GetSecretValue on it is granted solely to DevXWorkflowSecretsReader-SDKPython for the CI OIDC subjects — my ReadOnly role is denied by design. So the routing fix is verified structurally, and the first true green run for these three tests will be in CI.

Reviewer note

safety-gate does not apply here — this touches no workflow files, so the integration tests should run automatically on this PR.

The autoevals default judge model is gpt-5-mini. If the provisioned key's account lacks access to it, expect a model 404 rather than a 401; that would be a key-provisioning follow-up, not a code issue.

[agentcore-devx-automation]

Claude Security Review: no high-confidence findings. (run)

[jariy17]

Summary

Follow-up to #614, which took the evaluation group from 6 failures to 3. The three DeepEval tests now pass — confirming both the span-fixture fix and the OPENAI_API_KEY wiring work. The three remaining AutoEvalsAdapter tests fail with:

openai.AuthenticationError: Braintrust gateway error: auth failed
[401 Unauthorized]: Invalid API Key ... org: None

The key is valid — it was being sent to the wrong service.autoevals does not call OpenAI directly by default (autoevals/oai.py):

ifbase_urlisNone:
base_url=os.environ.get("OPENAI_BASE_URL") or_gateway_url() # https://gateway.braintrust.devif_is_gateway_url(base_url):
api_key=os.environ.get("BRAINTRUST_API_KEY") oros.environ.get("OPENAI_API_KEY")

With OPENAI_BASE_URL unset it routes to the Braintrust AI gateway, which rejects an OpenAI key. This is documented behaviour, not a bug — init()'s own docstring example pairs the gateway base_url with BRAINTRUST_API_KEY.

Fix

Pass an explicit client, which is the provider-selection path autoevals documents. Its api_key / base_url arguments are marked "Deprecated: Use the client argument", so this is the forward-compatible option rather than setting OPENAI_BASE_URL in CI.

The client is built once in a fixture; each test passes it to the scorer, keeping the provider choice visible at the call site.

Scoped to the Autoevals tests only — DeepEval is untouched and already passing.

Testing

# an injected client bypasses the gateway: probe with a deliberately invalid key
Error code: 401 - Incorrect API key provided: sk-fake-*robe ...
# ^ OpenAI's own error, no Braintrust in the message (previously: "Braintrust gateway error")
ruff format --check -> 1 file already formatted
ruff check -> All checks passed!

I could not run these tests against the real key locally: the value lives only in the central DevX Secrets Manager account, and secretsmanager:GetSecretValue on it is granted solely to DevXWorkflowSecretsReader-SDKPython for the CI OIDC subjects — my ReadOnly role is denied by design. So the routing fix is verified structurally, and the first true green run for these three tests will be in CI.

Reviewer note

safety-gate does not apply here — this touches no workflow files, so the integration tests should run automatically on this PR.

The autoevals default judge model is gpt-5-mini. If the provisioned key's account lacks access to it, expect a model 404 rather than a 401; that would be a key-provisioning follow-up, not a code issue.

[agentcore-devx-automation]

Claude Security Review: no high-confidence findings. (run)

[jariy17]

Summary

Follow-up to #614, which took the evaluation group from 6 failures to 3. The three DeepEval tests now pass — confirming both the span-fixture fix and the OPENAI_API_KEY wiring work. The three remaining AutoEvalsAdapter tests fail with:

openai.AuthenticationError: Braintrust gateway error: auth failed
[401 Unauthorized]: Invalid API Key ... org: None

The key is valid — it was being sent to the wrong service.autoevals does not call OpenAI directly by default (autoevals/oai.py):

ifbase_urlisNone:
base_url=os.environ.get("OPENAI_BASE_URL") or_gateway_url() # https://gateway.braintrust.devif_is_gateway_url(base_url):
api_key=os.environ.get("BRAINTRUST_API_KEY") oros.environ.get("OPENAI_API_KEY")

With OPENAI_BASE_URL unset it routes to the Braintrust AI gateway, which rejects an OpenAI key. This is documented behaviour, not a bug — init()'s own docstring example pairs the gateway base_url with BRAINTRUST_API_KEY.

Fix

Pass an explicit client, which is the provider-selection path autoevals documents. Its api_key / base_url arguments are marked "Deprecated: Use the client argument", so this is the forward-compatible option rather than setting OPENAI_BASE_URL in CI.

The client is built once in a fixture; each test passes it to the scorer, keeping the provider choice visible at the call site.

Scoped to the Autoevals tests only — DeepEval is untouched and already passing.

Testing

# an injected client bypasses the gateway: probe with a deliberately invalid key
Error code: 401 - Incorrect API key provided: sk-fake-*robe ...
# ^ OpenAI's own error, no Braintrust in the message (previously: "Braintrust gateway error")
ruff format --check -> 1 file already formatted
ruff check -> All checks passed!

I could not run these tests against the real key locally: the value lives only in the central DevX Secrets Manager account, and secretsmanager:GetSecretValue on it is granted solely to DevXWorkflowSecretsReader-SDKPython for the CI OIDC subjects — my ReadOnly role is denied by design. So the routing fix is verified structurally, and the first true green run for these three tests will be in CI.

Reviewer note

safety-gate does not apply here — this touches no workflow files, so the integration tests should run automatically on this PR.

The autoevals default judge model is gpt-5-mini. If the provisioned key's account lacks access to it, expect a model 404 rather than a 401; that would be a key-provisioning follow-up, not a code issue.

[agentcore-devx-automation]

Claude Security Review: no high-confidence findings. (run)

[jariy17]

Summary

Follow-up to #614, which took the evaluation group from 6 failures to 3. The three DeepEval tests now pass — confirming both the span-fixture fix and the OPENAI_API_KEY wiring work. The three remaining AutoEvalsAdapter tests fail with:

openai.AuthenticationError: Braintrust gateway error: auth failed
[401 Unauthorized]: Invalid API Key ... org: None

The key is valid — it was being sent to the wrong service.autoevals does not call OpenAI directly by default (autoevals/oai.py):

ifbase_urlisNone:
base_url=os.environ.get("OPENAI_BASE_URL") or_gateway_url() # https://gateway.braintrust.devif_is_gateway_url(base_url):
api_key=os.environ.get("BRAINTRUST_API_KEY") oros.environ.get("OPENAI_API_KEY")

With OPENAI_BASE_URL unset it routes to the Braintrust AI gateway, which rejects an OpenAI key. This is documented behaviour, not a bug — init()'s own docstring example pairs the gateway base_url with BRAINTRUST_API_KEY.

Fix

Pass an explicit client, which is the provider-selection path autoevals documents. Its api_key / base_url arguments are marked "Deprecated: Use the client argument", so this is the forward-compatible option rather than setting OPENAI_BASE_URL in CI.

The client is built once in a fixture; each test passes it to the scorer, keeping the provider choice visible at the call site.

Scoped to the Autoevals tests only — DeepEval is untouched and already passing.

Testing

# an injected client bypasses the gateway: probe with a deliberately invalid key
Error code: 401 - Incorrect API key provided: sk-fake-*robe ...
# ^ OpenAI's own error, no Braintrust in the message (previously: "Braintrust gateway error")
ruff format --check -> 1 file already formatted
ruff check -> All checks passed!

I could not run these tests against the real key locally: the value lives only in the central DevX Secrets Manager account, and secretsmanager:GetSecretValue on it is granted solely to DevXWorkflowSecretsReader-SDKPython for the CI OIDC subjects — my ReadOnly role is denied by design. So the routing fix is verified structurally, and the first true green run for these three tests will be in CI.

Reviewer note

safety-gate does not apply here — this touches no workflow files, so the integration tests should run automatically on this PR.

The autoevals default judge model is gpt-5-mini. If the provisioned key's account lacks access to it, expect a model 404 rather than a 401; that would be a key-provisioning follow-up, not a code issue.

[agentcore-devx-automation]

Claude Security Review: no high-confidence findings. (run)

[jariy17]

Summary

Follow-up to #614, which took the evaluation group from 6 failures to 3. The three DeepEval tests now pass — confirming both the span-fixture fix and the OPENAI_API_KEY wiring work. The three remaining AutoEvalsAdapter tests fail with:

openai.AuthenticationError: Braintrust gateway error: auth failed
[401 Unauthorized]: Invalid API Key ... org: None

The key is valid — it was being sent to the wrong service.autoevals does not call OpenAI directly by default (autoevals/oai.py):

ifbase_urlisNone:
base_url=os.environ.get("OPENAI_BASE_URL") or_gateway_url() # https://gateway.braintrust.devif_is_gateway_url(base_url):
api_key=os.environ.get("BRAINTRUST_API_KEY") oros.environ.get("OPENAI_API_KEY")

With OPENAI_BASE_URL unset it routes to the Braintrust AI gateway, which rejects an OpenAI key. This is documented behaviour, not a bug — init()'s own docstring example pairs the gateway base_url with BRAINTRUST_API_KEY.

Fix

Pass an explicit client, which is the provider-selection path autoevals documents. Its api_key / base_url arguments are marked "Deprecated: Use the client argument", so this is the forward-compatible option rather than setting OPENAI_BASE_URL in CI.

The client is built once in a fixture; each test passes it to the scorer, keeping the provider choice visible at the call site.

Scoped to the Autoevals tests only — DeepEval is untouched and already passing.

Testing

# an injected client bypasses the gateway: probe with a deliberately invalid key
Error code: 401 - Incorrect API key provided: sk-fake-*robe ...
# ^ OpenAI's own error, no Braintrust in the message (previously: "Braintrust gateway error")
ruff format --check -> 1 file already formatted
ruff check -> All checks passed!

I could not run these tests against the real key locally: the value lives only in the central DevX Secrets Manager account, and secretsmanager:GetSecretValue on it is granted solely to DevXWorkflowSecretsReader-SDKPython for the CI OIDC subjects — my ReadOnly role is denied by design. So the routing fix is verified structurally, and the first true green run for these three tests will be in CI.

Reviewer note

safety-gate does not apply here — this touches no workflow files, so the integration tests should run automatically on this PR.

The autoevals default judge model is gpt-5-mini. If the provisioned key's account lacks access to it, expect a model 404 rather than a 401; that would be a key-provisioning follow-up, not a code issue.

[agentcore-devx-automation]

Claude Security Review: no high-confidence findings. (run)

[jariy17]

Summary

Follow-up to #614, which took the evaluation group from 6 failures to 3. The three DeepEval tests now pass — confirming both the span-fixture fix and the OPENAI_API_KEY wiring work. The three remaining AutoEvalsAdapter tests fail with:

openai.AuthenticationError: Braintrust gateway error: auth failed
[401 Unauthorized]: Invalid API Key ... org: None

The key is valid — it was being sent to the wrong service.autoevals does not call OpenAI directly by default (autoevals/oai.py):

ifbase_urlisNone:
base_url=os.environ.get("OPENAI_BASE_URL") or_gateway_url() # https://gateway.braintrust.devif_is_gateway_url(base_url):
api_key=os.environ.get("BRAINTRUST_API_KEY") oros.environ.get("OPENAI_API_KEY")

With OPENAI_BASE_URL unset it routes to the Braintrust AI gateway, which rejects an OpenAI key. This is documented behaviour, not a bug — init()'s own docstring example pairs the gateway base_url with BRAINTRUST_API_KEY.

Fix

Pass an explicit client, which is the provider-selection path autoevals documents. Its api_key / base_url arguments are marked "Deprecated: Use the client argument", so this is the forward-compatible option rather than setting OPENAI_BASE_URL in CI.

The client is built once in a fixture; each test passes it to the scorer, keeping the provider choice visible at the call site.

Scoped to the Autoevals tests only — DeepEval is untouched and already passing.

Testing

# an injected client bypasses the gateway: probe with a deliberately invalid key
Error code: 401 - Incorrect API key provided: sk-fake-*robe ...
# ^ OpenAI's own error, no Braintrust in the message (previously: "Braintrust gateway error")
ruff format --check -> 1 file already formatted
ruff check -> All checks passed!

I could not run these tests against the real key locally: the value lives only in the central DevX Secrets Manager account, and secretsmanager:GetSecretValue on it is granted solely to DevXWorkflowSecretsReader-SDKPython for the CI OIDC subjects — my ReadOnly role is denied by design. So the routing fix is verified structurally, and the first true green run for these three tests will be in CI.

Reviewer note

safety-gate does not apply here — this touches no workflow files, so the integration tests should run automatically on this PR.

The autoevals default judge model is gpt-5-mini. If the provisioned key's account lacks access to it, expect a model 404 rather than a 401; that would be a key-provisioning follow-up, not a code issue.

[agentcore-devx-automation]

Claude Security Review: no high-confidence findings. (run)

[jariy17]

Summary

Follow-up to #614, which took the evaluation group from 6 failures to 3. The three DeepEval tests now pass — confirming both the span-fixture fix and the OPENAI_API_KEY wiring work. The three remaining AutoEvalsAdapter tests fail with:

openai.AuthenticationError: Braintrust gateway error: auth failed
[401 Unauthorized]: Invalid API Key ... org: None

The key is valid — it was being sent to the wrong service.autoevals does not call OpenAI directly by default (autoevals/oai.py):

ifbase_urlisNone:
base_url=os.environ.get("OPENAI_BASE_URL") or_gateway_url() # https://gateway.braintrust.devif_is_gateway_url(base_url):
api_key=os.environ.get("BRAINTRUST_API_KEY") oros.environ.get("OPENAI_API_KEY")

With OPENAI_BASE_URL unset it routes to the Braintrust AI gateway, which rejects an OpenAI key. This is documented behaviour, not a bug — init()'s own docstring example pairs the gateway base_url with BRAINTRUST_API_KEY.

Fix

Pass an explicit client, which is the provider-selection path autoevals documents. Its api_key / base_url arguments are marked "Deprecated: Use the client argument", so this is the forward-compatible option rather than setting OPENAI_BASE_URL in CI.

The client is built once in a fixture; each test passes it to the scorer, keeping the provider choice visible at the call site.

Scoped to the Autoevals tests only — DeepEval is untouched and already passing.

Testing

# an injected client bypasses the gateway: probe with a deliberately invalid key
Error code: 401 - Incorrect API key provided: sk-fake-*robe ...
# ^ OpenAI's own error, no Braintrust in the message (previously: "Braintrust gateway error")
ruff format --check -> 1 file already formatted
ruff check -> All checks passed!

I could not run these tests against the real key locally: the value lives only in the central DevX Secrets Manager account, and secretsmanager:GetSecretValue on it is granted solely to DevXWorkflowSecretsReader-SDKPython for the CI OIDC subjects — my ReadOnly role is denied by design. So the routing fix is verified structurally, and the first true green run for these three tests will be in CI.

Reviewer note

safety-gate does not apply here — this touches no workflow files, so the integration tests should run automatically on this PR.

The autoevals default judge model is gpt-5-mini. If the provisioned key's account lacks access to it, expect a model 404 rather than a 401; that would be a key-provisioning follow-up, not a code issue.

[agentcore-devx-automation]

Claude Security Review: no high-confidence findings. (run)

[jariy17]

Summary

Follow-up to #614, which took the evaluation group from 6 failures to 3. The three DeepEval tests now pass — confirming both the span-fixture fix and the OPENAI_API_KEY wiring work. The three remaining AutoEvalsAdapter tests fail with:

openai.AuthenticationError: Braintrust gateway error: auth failed
[401 Unauthorized]: Invalid API Key ... org: None

The key is valid — it was being sent to the wrong service.autoevals does not call OpenAI directly by default (autoevals/oai.py):

ifbase_urlisNone:
base_url=os.environ.get("OPENAI_BASE_URL") or_gateway_url() # https://gateway.braintrust.devif_is_gateway_url(base_url):
api_key=os.environ.get("BRAINTRUST_API_KEY") oros.environ.get("OPENAI_API_KEY")

With OPENAI_BASE_URL unset it routes to the Braintrust AI gateway, which rejects an OpenAI key. This is documented behaviour, not a bug — init()'s own docstring example pairs the gateway base_url with BRAINTRUST_API_KEY.

Fix

Pass an explicit client, which is the provider-selection path autoevals documents. Its api_key / base_url arguments are marked "Deprecated: Use the client argument", so this is the forward-compatible option rather than setting OPENAI_BASE_URL in CI.

The client is built once in a fixture; each test passes it to the scorer, keeping the provider choice visible at the call site.

Scoped to the Autoevals tests only — DeepEval is untouched and already passing.

Testing

# an injected client bypasses the gateway: probe with a deliberately invalid key
Error code: 401 - Incorrect API key provided: sk-fake-*robe ...
# ^ OpenAI's own error, no Braintrust in the message (previously: "Braintrust gateway error")
ruff format --check -> 1 file already formatted
ruff check -> All checks passed!

I could not run these tests against the real key locally: the value lives only in the central DevX Secrets Manager account, and secretsmanager:GetSecretValue on it is granted solely to DevXWorkflowSecretsReader-SDKPython for the CI OIDC subjects — my ReadOnly role is denied by design. So the routing fix is verified structurally, and the first true green run for these three tests will be in CI.

Reviewer note

safety-gate does not apply here — this touches no workflow files, so the integration tests should run automatically on this PR.

The autoevals default judge model is gpt-5-mini. If the provisioned key's account lacks access to it, expect a model 404 rather than a 401; that would be a key-provisioning follow-up, not a code issue.

[agentcore-devx-automation]

Claude Security Review: no high-confidence findings. (run)