feat(evaluators): add AgentControl support for Orbit preset SLM scorer invocation

[namrataghadi-galileo]

Summary

• Adds AgentControl support for Orbit preset SLM scorer invocation and is paired with rungalileo/orbit#1720.
• Requires scorer_id; scorer_label is optional display metadata and scorer_version_id is a deprecated compatibility identifier.
• Calls /api/v1/scorers/invoke with a scoped internal HS256 JWT (scorers.invoke).
• Dual-writes legacy inputs and a structured record during the rollout, including selector-aligned input/output plus Step context, tools, and ground truth.
• Sends an Orbit execution timeout below the AgentControl HTTP deadline (8 seconds for the default 10-second deadline).

Scope

• Orbit owns feature-flag routing between RuntimeScorer and the legacy adapter.
• Nine flat preset SLM scorers are validated for direct runtime execution.
• Existing Step context propagation through models, engine, integrations, and SDKs is retained.
• Broader preset SLM runtime coverage is deferred to Jira phases 2 and 3.
• Customer-owned Luna scorers are not supported in this release.
• No backend selector, Protect payload construction, Wizard configuration, trusted tenant identity, or Galileo record-tree construction is added.

Risk and Rollout

• Risk level: medium.
• Deploy Orbit #1720 and keep its legacy adapter available for the tandem rollout.
• Roll back AgentControl to the legacy inputs.query/inputs.response contract if needed.
• Retain dual-write until older supported Orbit versions are retired.

Testing

• Galileo evaluator tests (116 passed)
• Models tests (71 passed)
• Engine tests (117 passed)
• Affected Python SDK tests (103 passed)
• Built-in evaluator tests (310 passed)
• TypeScript SDK tests, lint, and typecheck
• Repository lint, mypy, and git diff --check
• Full Python SDK target requires local PostgreSQL on localhost:5432

Follow-up

Orbit #1720 must update all six AgentControl git references to commit 8a2b5d52717090ed7c65b627035f73d89c46f5ff and regenerate requirements before the tandem merge.

[codecov]

Codecov Report

❌ Patch coverage is 99.25373% with 1 line in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...src/agent_control_evaluator_galileo/luna/client.py	96.96%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[abhinav-galileo]

[P1][pre-existing] Make cached async clients event-loop-safe

Cached evaluator instances retain pooled httpx.AsyncClient objects, while the synchronous decorator creates a new event loop per call. Reusing a keep-alive connection on a later loop can fail with RuntimeError: Event loop is closed, and cache eviction does not close these clients. This predates this PR and is not a regression introduced here; please track loop-scoped ownership or a persistent-loop lifecycle with awaited cleanup separately.

[namrataghadi-galileo]

@abhinav-galileo Review follow-up for 8a2b5d5:

• Structured record: addressed using Orbit #1720 as the authoritative contract. Orbit rejects conflicts between legacy inputs.query/inputs.response and record.input/record.output, so selector-selected values are intentionally dual-written. The full Step still supplies context, tools, dataset output, type, and name; the contract test verifies those structured fields are preserved.
• Scorer version identity: addressed. Metadata now uses requested_scorer_version_id, and docs state that Orbit executes the scorer's current default version.
• Timeout ordering: addressed. The server timeout is 80% of the HTTP deadline, and equal/greater explicit values fail locally before any request. Tests cover all branches.

The dependency-floor, invocation-resolved ADK tools, forward-compatible config, and deep Step snapshot threads are not addressed in this scoped change and remain open. Config strictness currently mirrors Orbit #1720's extra="forbid" contract.

[namrataghadi-galileo]

@abhinav-galileo Follow-up on the four remaining review threads:

• Dependency floors (#discussion_r3865624436): addressed in bdd30d5. The coordinated 8.6.0 floors now cover engine models/evaluators, builtin models and the Galileo extra, Galileo models/evaluators, and SDK/server runtime evaluators. evaluate_with_context(data, step) remains the additive hook, so evaluators implementing only evaluate(data) stay compatible.
• Invocation-resolved ADK tools (#discussion_r3865624464): addressed in bdd30d5. Model callbacks now prefer the exact current LlmRequest.tools_dict, treat an empty resolved map as authoritative, fall back to the bind-time registry only for older ADK versions, and reuse the captured set for the matching post-model evaluation. Tests cover dynamic override and an empty invocation set.
• Forward-compatible scorer config (#discussion_r3865624472): no code change is needed for this rollout. Orbit #1720 is authoritative and defines extra="forbid" with only threshold, score_threshold, and request_timeout_seconds. AgentControl intentionally mirrors that allowlist and should extend it in lockstep with Orbit.
• Deep Step snapshots (#discussion_r3865624478): no code change is needed for the scorer-invoke rollout. Luna only reads the trusted request-scoped Step. True evaluator isolation would require copying both selector-selected data and the full Step per evaluator in the hot path, so that should be a separate design/performance change if we decide to guarantee deep isolation.

[namrataghadi-galileo]

@abhinav-galileo Follow-up on the four remaining review threads:

• Dependency floors (#discussion_r3865624436): addressed in bdd30d5. The coordinated 8.6.0 floors now cover engine models/evaluators, builtin models and the Galileo extra, Galileo models/evaluators, and SDK/server runtime evaluators. evaluate_with_context(data, step) remains the additive hook, so evaluators implementing only evaluate(data) stay compatible.
• Invocation-resolved ADK tools (#discussion_r3865624464): addressed in bdd30d5. Model callbacks now prefer the exact current LlmRequest.tools_dict, treat an empty resolved map as authoritative, fall back to the bind-time registry only for older ADK versions, and reuse the captured set for the matching post-model evaluation. Tests cover dynamic override and an empty invocation set.
• Forward-compatible scorer config (#discussion_r3865624472): no code change is needed for this rollout. Orbit #1720 is authoritative and defines extra="forbid" with only threshold, score_threshold, and request_timeout_seconds. AgentControl intentionally mirrors that allowlist and should extend it in lockstep with Orbit.
• Deep Step snapshots (#discussion_r3865624478): no code change is needed for the scorer-invoke rollout. Luna only reads the trusted request-scoped Step. True evaluator isolation would require copying both selector-selected data and the full Step per evaluator in the hot path, so that should be a separate design/performance change if we decide to guarantee deep isolation.

[namrataghadi-galileo]

Summary

• Adds AgentControl support for Orbit preset SLM scorer invocation and is paired with rungalileo/orbit#1720.
• Requires scorer_id; scorer_label is optional display metadata and scorer_version_id is a deprecated compatibility identifier.
• Calls /api/v1/scorers/invoke with a scoped internal HS256 JWT (scorers.invoke).
• Dual-writes legacy inputs and a structured record during the rollout, including selector-aligned input/output plus Step context, tools, and ground truth.
• Sends an Orbit execution timeout below the AgentControl HTTP deadline (8 seconds for the default 10-second deadline).

Scope

• Orbit owns feature-flag routing between RuntimeScorer and the legacy adapter.
• Nine flat preset SLM scorers are validated for direct runtime execution.
• Existing Step context propagation through models, engine, integrations, and SDKs is retained.
• Broader preset SLM runtime coverage is deferred to Jira phases 2 and 3.
• Customer-owned Luna scorers are not supported in this release.
• No backend selector, Protect payload construction, Wizard configuration, trusted tenant identity, or Galileo record-tree construction is added.

Risk and Rollout

• Risk level: medium.
• Deploy Orbit #1720 and keep its legacy adapter available for the tandem rollout.
• Roll back AgentControl to the legacy inputs.query/inputs.response contract if needed.
• Retain dual-write until older supported Orbit versions are retired.

Testing

• Galileo evaluator tests (116 passed)
• Models tests (71 passed)
• Engine tests (117 passed)
• Affected Python SDK tests (103 passed)
• Built-in evaluator tests (310 passed)
• TypeScript SDK tests, lint, and typecheck
• Repository lint, mypy, and git diff --check
• Full Python SDK target requires local PostgreSQL on localhost:5432

Follow-up

Orbit #1720 must update all six AgentControl git references to commit 8a2b5d52717090ed7c65b627035f73d89c46f5ff and regenerate requirements before the tandem merge.

[codecov]

Codecov Report

❌ Patch coverage is 99.25373% with 1 line in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...src/agent_control_evaluator_galileo/luna/client.py	96.96%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[abhinav-galileo]

[P1][pre-existing] Make cached async clients event-loop-safe

Cached evaluator instances retain pooled httpx.AsyncClient objects, while the synchronous decorator creates a new event loop per call. Reusing a keep-alive connection on a later loop can fail with RuntimeError: Event loop is closed, and cache eviction does not close these clients. This predates this PR and is not a regression introduced here; please track loop-scoped ownership or a persistent-loop lifecycle with awaited cleanup separately.

[namrataghadi-galileo]

@abhinav-galileo Review follow-up for 8a2b5d5:

• Structured record: addressed using Orbit #1720 as the authoritative contract. Orbit rejects conflicts between legacy inputs.query/inputs.response and record.input/record.output, so selector-selected values are intentionally dual-written. The full Step still supplies context, tools, dataset output, type, and name; the contract test verifies those structured fields are preserved.
• Scorer version identity: addressed. Metadata now uses requested_scorer_version_id, and docs state that Orbit executes the scorer's current default version.
• Timeout ordering: addressed. The server timeout is 80% of the HTTP deadline, and equal/greater explicit values fail locally before any request. Tests cover all branches.

The dependency-floor, invocation-resolved ADK tools, forward-compatible config, and deep Step snapshot threads are not addressed in this scoped change and remain open. Config strictness currently mirrors Orbit #1720's extra="forbid" contract.

[namrataghadi-galileo]

@abhinav-galileo Follow-up on the four remaining review threads:

• Dependency floors (#discussion_r3865624436): addressed in bdd30d5. The coordinated 8.6.0 floors now cover engine models/evaluators, builtin models and the Galileo extra, Galileo models/evaluators, and SDK/server runtime evaluators. evaluate_with_context(data, step) remains the additive hook, so evaluators implementing only evaluate(data) stay compatible.
• Invocation-resolved ADK tools (#discussion_r3865624464): addressed in bdd30d5. Model callbacks now prefer the exact current LlmRequest.tools_dict, treat an empty resolved map as authoritative, fall back to the bind-time registry only for older ADK versions, and reuse the captured set for the matching post-model evaluation. Tests cover dynamic override and an empty invocation set.
• Forward-compatible scorer config (#discussion_r3865624472): no code change is needed for this rollout. Orbit #1720 is authoritative and defines extra="forbid" with only threshold, score_threshold, and request_timeout_seconds. AgentControl intentionally mirrors that allowlist and should extend it in lockstep with Orbit.
• Deep Step snapshots (#discussion_r3865624478): no code change is needed for the scorer-invoke rollout. Luna only reads the trusted request-scoped Step. True evaluator isolation would require copying both selector-selected data and the full Step per evaluator in the hot path, so that should be a separate design/performance change if we decide to guarantee deep isolation.

[namrataghadi-galileo]

@abhinav-galileo Follow-up on the four remaining review threads:

• Dependency floors (#discussion_r3865624436): addressed in bdd30d5. The coordinated 8.6.0 floors now cover engine models/evaluators, builtin models and the Galileo extra, Galileo models/evaluators, and SDK/server runtime evaluators. evaluate_with_context(data, step) remains the additive hook, so evaluators implementing only evaluate(data) stay compatible.
• Invocation-resolved ADK tools (#discussion_r3865624464): addressed in bdd30d5. Model callbacks now prefer the exact current LlmRequest.tools_dict, treat an empty resolved map as authoritative, fall back to the bind-time registry only for older ADK versions, and reuse the captured set for the matching post-model evaluation. Tests cover dynamic override and an empty invocation set.
• Forward-compatible scorer config (#discussion_r3865624472): no code change is needed for this rollout. Orbit #1720 is authoritative and defines extra="forbid" with only threshold, score_threshold, and request_timeout_seconds. AgentControl intentionally mirrors that allowlist and should extend it in lockstep with Orbit.
• Deep Step snapshots (#discussion_r3865624478): no code change is needed for the scorer-invoke rollout. Luna only reads the trusted request-scoped Step. True evaluator isolation would require copying both selector-selected data and the full Step per evaluator in the hot path, so that should be a separate design/performance change if we decide to guarantee deep isolation.

[namrataghadi-galileo]

Summary

• Adds AgentControl support for Orbit preset SLM scorer invocation and is paired with rungalileo/orbit#1720.
• Requires scorer_id; scorer_label is optional display metadata and scorer_version_id is a deprecated compatibility identifier.
• Calls /api/v1/scorers/invoke with a scoped internal HS256 JWT (scorers.invoke).
• Dual-writes legacy inputs and a structured record during the rollout, including selector-aligned input/output plus Step context, tools, and ground truth.
• Sends an Orbit execution timeout below the AgentControl HTTP deadline (8 seconds for the default 10-second deadline).

Scope

• Orbit owns feature-flag routing between RuntimeScorer and the legacy adapter.
• Nine flat preset SLM scorers are validated for direct runtime execution.
• Existing Step context propagation through models, engine, integrations, and SDKs is retained.
• Broader preset SLM runtime coverage is deferred to Jira phases 2 and 3.
• Customer-owned Luna scorers are not supported in this release.
• No backend selector, Protect payload construction, Wizard configuration, trusted tenant identity, or Galileo record-tree construction is added.

Risk and Rollout

• Risk level: medium.
• Deploy Orbit #1720 and keep its legacy adapter available for the tandem rollout.
• Roll back AgentControl to the legacy inputs.query/inputs.response contract if needed.
• Retain dual-write until older supported Orbit versions are retired.

Testing

• Galileo evaluator tests (116 passed)
• Models tests (71 passed)
• Engine tests (117 passed)
• Affected Python SDK tests (103 passed)
• Built-in evaluator tests (310 passed)
• TypeScript SDK tests, lint, and typecheck
• Repository lint, mypy, and git diff --check
• Full Python SDK target requires local PostgreSQL on localhost:5432

Follow-up

Orbit #1720 must update all six AgentControl git references to commit 8a2b5d52717090ed7c65b627035f73d89c46f5ff and regenerate requirements before the tandem merge.

[codecov]

Codecov Report

❌ Patch coverage is 99.25373% with 1 line in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...src/agent_control_evaluator_galileo/luna/client.py	96.96%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[abhinav-galileo]

[P1][pre-existing] Make cached async clients event-loop-safe

Cached evaluator instances retain pooled httpx.AsyncClient objects, while the synchronous decorator creates a new event loop per call. Reusing a keep-alive connection on a later loop can fail with RuntimeError: Event loop is closed, and cache eviction does not close these clients. This predates this PR and is not a regression introduced here; please track loop-scoped ownership or a persistent-loop lifecycle with awaited cleanup separately.

[namrataghadi-galileo]

@abhinav-galileo Review follow-up for 8a2b5d5:

• Structured record: addressed using Orbit #1720 as the authoritative contract. Orbit rejects conflicts between legacy inputs.query/inputs.response and record.input/record.output, so selector-selected values are intentionally dual-written. The full Step still supplies context, tools, dataset output, type, and name; the contract test verifies those structured fields are preserved.
• Scorer version identity: addressed. Metadata now uses requested_scorer_version_id, and docs state that Orbit executes the scorer's current default version.
• Timeout ordering: addressed. The server timeout is 80% of the HTTP deadline, and equal/greater explicit values fail locally before any request. Tests cover all branches.

The dependency-floor, invocation-resolved ADK tools, forward-compatible config, and deep Step snapshot threads are not addressed in this scoped change and remain open. Config strictness currently mirrors Orbit #1720's extra="forbid" contract.

[namrataghadi-galileo]

@abhinav-galileo Follow-up on the four remaining review threads:

• Dependency floors (#discussion_r3865624436): addressed in bdd30d5. The coordinated 8.6.0 floors now cover engine models/evaluators, builtin models and the Galileo extra, Galileo models/evaluators, and SDK/server runtime evaluators. evaluate_with_context(data, step) remains the additive hook, so evaluators implementing only evaluate(data) stay compatible.
• Invocation-resolved ADK tools (#discussion_r3865624464): addressed in bdd30d5. Model callbacks now prefer the exact current LlmRequest.tools_dict, treat an empty resolved map as authoritative, fall back to the bind-time registry only for older ADK versions, and reuse the captured set for the matching post-model evaluation. Tests cover dynamic override and an empty invocation set.
• Forward-compatible scorer config (#discussion_r3865624472): no code change is needed for this rollout. Orbit #1720 is authoritative and defines extra="forbid" with only threshold, score_threshold, and request_timeout_seconds. AgentControl intentionally mirrors that allowlist and should extend it in lockstep with Orbit.
• Deep Step snapshots (#discussion_r3865624478): no code change is needed for the scorer-invoke rollout. Luna only reads the trusted request-scoped Step. True evaluator isolation would require copying both selector-selected data and the full Step per evaluator in the hot path, so that should be a separate design/performance change if we decide to guarantee deep isolation.

[namrataghadi-galileo]

@abhinav-galileo Follow-up on the four remaining review threads:

• Dependency floors (#discussion_r3865624436): addressed in bdd30d5. The coordinated 8.6.0 floors now cover engine models/evaluators, builtin models and the Galileo extra, Galileo models/evaluators, and SDK/server runtime evaluators. evaluate_with_context(data, step) remains the additive hook, so evaluators implementing only evaluate(data) stay compatible.
• Invocation-resolved ADK tools (#discussion_r3865624464): addressed in bdd30d5. Model callbacks now prefer the exact current LlmRequest.tools_dict, treat an empty resolved map as authoritative, fall back to the bind-time registry only for older ADK versions, and reuse the captured set for the matching post-model evaluation. Tests cover dynamic override and an empty invocation set.
• Forward-compatible scorer config (#discussion_r3865624472): no code change is needed for this rollout. Orbit #1720 is authoritative and defines extra="forbid" with only threshold, score_threshold, and request_timeout_seconds. AgentControl intentionally mirrors that allowlist and should extend it in lockstep with Orbit.
• Deep Step snapshots (#discussion_r3865624478): no code change is needed for the scorer-invoke rollout. Luna only reads the trusted request-scoped Step. True evaluator isolation would require copying both selector-selected data and the full Step per evaluator in the hot path, so that should be a separate design/performance change if we decide to guarantee deep isolation.

[namrataghadi-galileo]

Summary

• Adds AgentControl support for Orbit preset SLM scorer invocation and is paired with rungalileo/orbit#1720.
• Requires scorer_id; scorer_label is optional display metadata and scorer_version_id is a deprecated compatibility identifier.
• Calls /api/v1/scorers/invoke with a scoped internal HS256 JWT (scorers.invoke).
• Dual-writes legacy inputs and a structured record during the rollout, including selector-aligned input/output plus Step context, tools, and ground truth.
• Sends an Orbit execution timeout below the AgentControl HTTP deadline (8 seconds for the default 10-second deadline).

Scope

• Orbit owns feature-flag routing between RuntimeScorer and the legacy adapter.
• Nine flat preset SLM scorers are validated for direct runtime execution.
• Existing Step context propagation through models, engine, integrations, and SDKs is retained.
• Broader preset SLM runtime coverage is deferred to Jira phases 2 and 3.
• Customer-owned Luna scorers are not supported in this release.
• No backend selector, Protect payload construction, Wizard configuration, trusted tenant identity, or Galileo record-tree construction is added.

Risk and Rollout

• Risk level: medium.
• Deploy Orbit #1720 and keep its legacy adapter available for the tandem rollout.
• Roll back AgentControl to the legacy inputs.query/inputs.response contract if needed.
• Retain dual-write until older supported Orbit versions are retired.

Testing

• Galileo evaluator tests (116 passed)
• Models tests (71 passed)
• Engine tests (117 passed)
• Affected Python SDK tests (103 passed)
• Built-in evaluator tests (310 passed)
• TypeScript SDK tests, lint, and typecheck
• Repository lint, mypy, and git diff --check
• Full Python SDK target requires local PostgreSQL on localhost:5432

Follow-up

Orbit #1720 must update all six AgentControl git references to commit 8a2b5d52717090ed7c65b627035f73d89c46f5ff and regenerate requirements before the tandem merge.

[codecov]

Codecov Report

❌ Patch coverage is 99.25373% with 1 line in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...src/agent_control_evaluator_galileo/luna/client.py	96.96%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[abhinav-galileo]

[P1][pre-existing] Make cached async clients event-loop-safe

Cached evaluator instances retain pooled httpx.AsyncClient objects, while the synchronous decorator creates a new event loop per call. Reusing a keep-alive connection on a later loop can fail with RuntimeError: Event loop is closed, and cache eviction does not close these clients. This predates this PR and is not a regression introduced here; please track loop-scoped ownership or a persistent-loop lifecycle with awaited cleanup separately.

[namrataghadi-galileo]

@abhinav-galileo Review follow-up for 8a2b5d5:

• Structured record: addressed using Orbit #1720 as the authoritative contract. Orbit rejects conflicts between legacy inputs.query/inputs.response and record.input/record.output, so selector-selected values are intentionally dual-written. The full Step still supplies context, tools, dataset output, type, and name; the contract test verifies those structured fields are preserved.
• Scorer version identity: addressed. Metadata now uses requested_scorer_version_id, and docs state that Orbit executes the scorer's current default version.
• Timeout ordering: addressed. The server timeout is 80% of the HTTP deadline, and equal/greater explicit values fail locally before any request. Tests cover all branches.

The dependency-floor, invocation-resolved ADK tools, forward-compatible config, and deep Step snapshot threads are not addressed in this scoped change and remain open. Config strictness currently mirrors Orbit #1720's extra="forbid" contract.

[namrataghadi-galileo]

@abhinav-galileo Follow-up on the four remaining review threads:

• Dependency floors (#discussion_r3865624436): addressed in bdd30d5. The coordinated 8.6.0 floors now cover engine models/evaluators, builtin models and the Galileo extra, Galileo models/evaluators, and SDK/server runtime evaluators. evaluate_with_context(data, step) remains the additive hook, so evaluators implementing only evaluate(data) stay compatible.
• Invocation-resolved ADK tools (#discussion_r3865624464): addressed in bdd30d5. Model callbacks now prefer the exact current LlmRequest.tools_dict, treat an empty resolved map as authoritative, fall back to the bind-time registry only for older ADK versions, and reuse the captured set for the matching post-model evaluation. Tests cover dynamic override and an empty invocation set.
• Forward-compatible scorer config (#discussion_r3865624472): no code change is needed for this rollout. Orbit #1720 is authoritative and defines extra="forbid" with only threshold, score_threshold, and request_timeout_seconds. AgentControl intentionally mirrors that allowlist and should extend it in lockstep with Orbit.
• Deep Step snapshots (#discussion_r3865624478): no code change is needed for the scorer-invoke rollout. Luna only reads the trusted request-scoped Step. True evaluator isolation would require copying both selector-selected data and the full Step per evaluator in the hot path, so that should be a separate design/performance change if we decide to guarantee deep isolation.

[namrataghadi-galileo]

@abhinav-galileo Follow-up on the four remaining review threads:

• Dependency floors (#discussion_r3865624436): addressed in bdd30d5. The coordinated 8.6.0 floors now cover engine models/evaluators, builtin models and the Galileo extra, Galileo models/evaluators, and SDK/server runtime evaluators. evaluate_with_context(data, step) remains the additive hook, so evaluators implementing only evaluate(data) stay compatible.
• Invocation-resolved ADK tools (#discussion_r3865624464): addressed in bdd30d5. Model callbacks now prefer the exact current LlmRequest.tools_dict, treat an empty resolved map as authoritative, fall back to the bind-time registry only for older ADK versions, and reuse the captured set for the matching post-model evaluation. Tests cover dynamic override and an empty invocation set.
• Forward-compatible scorer config (#discussion_r3865624472): no code change is needed for this rollout. Orbit #1720 is authoritative and defines extra="forbid" with only threshold, score_threshold, and request_timeout_seconds. AgentControl intentionally mirrors that allowlist and should extend it in lockstep with Orbit.
• Deep Step snapshots (#discussion_r3865624478): no code change is needed for the scorer-invoke rollout. Luna only reads the trusted request-scoped Step. True evaluator isolation would require copying both selector-selected data and the full Step per evaluator in the hot path, so that should be a separate design/performance change if we decide to guarantee deep isolation.

[namrataghadi-galileo]

Summary

• Adds AgentControl support for Orbit preset SLM scorer invocation and is paired with rungalileo/orbit#1720.
• Requires scorer_id; scorer_label is optional display metadata and scorer_version_id is a deprecated compatibility identifier.
• Calls /api/v1/scorers/invoke with a scoped internal HS256 JWT (scorers.invoke).
• Dual-writes legacy inputs and a structured record during the rollout, including selector-aligned input/output plus Step context, tools, and ground truth.
• Sends an Orbit execution timeout below the AgentControl HTTP deadline (8 seconds for the default 10-second deadline).

Scope

• Orbit owns feature-flag routing between RuntimeScorer and the legacy adapter.
• Nine flat preset SLM scorers are validated for direct runtime execution.
• Existing Step context propagation through models, engine, integrations, and SDKs is retained.
• Broader preset SLM runtime coverage is deferred to Jira phases 2 and 3.
• Customer-owned Luna scorers are not supported in this release.
• No backend selector, Protect payload construction, Wizard configuration, trusted tenant identity, or Galileo record-tree construction is added.

Risk and Rollout

• Risk level: medium.
• Deploy Orbit #1720 and keep its legacy adapter available for the tandem rollout.
• Roll back AgentControl to the legacy inputs.query/inputs.response contract if needed.
• Retain dual-write until older supported Orbit versions are retired.

Testing

• Galileo evaluator tests (116 passed)
• Models tests (71 passed)
• Engine tests (117 passed)
• Affected Python SDK tests (103 passed)
• Built-in evaluator tests (310 passed)
• TypeScript SDK tests, lint, and typecheck
• Repository lint, mypy, and git diff --check
• Full Python SDK target requires local PostgreSQL on localhost:5432

Follow-up

Orbit #1720 must update all six AgentControl git references to commit 8a2b5d52717090ed7c65b627035f73d89c46f5ff and regenerate requirements before the tandem merge.

[codecov]

Codecov Report

❌ Patch coverage is 99.25373% with 1 line in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...src/agent_control_evaluator_galileo/luna/client.py	96.96%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[abhinav-galileo]

[P1][pre-existing] Make cached async clients event-loop-safe

Cached evaluator instances retain pooled httpx.AsyncClient objects, while the synchronous decorator creates a new event loop per call. Reusing a keep-alive connection on a later loop can fail with RuntimeError: Event loop is closed, and cache eviction does not close these clients. This predates this PR and is not a regression introduced here; please track loop-scoped ownership or a persistent-loop lifecycle with awaited cleanup separately.

[namrataghadi-galileo]

@abhinav-galileo Review follow-up for 8a2b5d5:

• Structured record: addressed using Orbit #1720 as the authoritative contract. Orbit rejects conflicts between legacy inputs.query/inputs.response and record.input/record.output, so selector-selected values are intentionally dual-written. The full Step still supplies context, tools, dataset output, type, and name; the contract test verifies those structured fields are preserved.
• Scorer version identity: addressed. Metadata now uses requested_scorer_version_id, and docs state that Orbit executes the scorer's current default version.
• Timeout ordering: addressed. The server timeout is 80% of the HTTP deadline, and equal/greater explicit values fail locally before any request. Tests cover all branches.

The dependency-floor, invocation-resolved ADK tools, forward-compatible config, and deep Step snapshot threads are not addressed in this scoped change and remain open. Config strictness currently mirrors Orbit #1720's extra="forbid" contract.

[namrataghadi-galileo]

@abhinav-galileo Follow-up on the four remaining review threads:

• Dependency floors (#discussion_r3865624436): addressed in bdd30d5. The coordinated 8.6.0 floors now cover engine models/evaluators, builtin models and the Galileo extra, Galileo models/evaluators, and SDK/server runtime evaluators. evaluate_with_context(data, step) remains the additive hook, so evaluators implementing only evaluate(data) stay compatible.
• Invocation-resolved ADK tools (#discussion_r3865624464): addressed in bdd30d5. Model callbacks now prefer the exact current LlmRequest.tools_dict, treat an empty resolved map as authoritative, fall back to the bind-time registry only for older ADK versions, and reuse the captured set for the matching post-model evaluation. Tests cover dynamic override and an empty invocation set.
• Forward-compatible scorer config (#discussion_r3865624472): no code change is needed for this rollout. Orbit #1720 is authoritative and defines extra="forbid" with only threshold, score_threshold, and request_timeout_seconds. AgentControl intentionally mirrors that allowlist and should extend it in lockstep with Orbit.
• Deep Step snapshots (#discussion_r3865624478): no code change is needed for the scorer-invoke rollout. Luna only reads the trusted request-scoped Step. True evaluator isolation would require copying both selector-selected data and the full Step per evaluator in the hot path, so that should be a separate design/performance change if we decide to guarantee deep isolation.

[namrataghadi-galileo]

@abhinav-galileo Follow-up on the four remaining review threads:

• Dependency floors (#discussion_r3865624436): addressed in bdd30d5. The coordinated 8.6.0 floors now cover engine models/evaluators, builtin models and the Galileo extra, Galileo models/evaluators, and SDK/server runtime evaluators. evaluate_with_context(data, step) remains the additive hook, so evaluators implementing only evaluate(data) stay compatible.
• Invocation-resolved ADK tools (#discussion_r3865624464): addressed in bdd30d5. Model callbacks now prefer the exact current LlmRequest.tools_dict, treat an empty resolved map as authoritative, fall back to the bind-time registry only for older ADK versions, and reuse the captured set for the matching post-model evaluation. Tests cover dynamic override and an empty invocation set.
• Forward-compatible scorer config (#discussion_r3865624472): no code change is needed for this rollout. Orbit #1720 is authoritative and defines extra="forbid" with only threshold, score_threshold, and request_timeout_seconds. AgentControl intentionally mirrors that allowlist and should extend it in lockstep with Orbit.
• Deep Step snapshots (#discussion_r3865624478): no code change is needed for the scorer-invoke rollout. Luna only reads the trusted request-scoped Step. True evaluator isolation would require copying both selector-selected data and the full Step per evaluator in the hot path, so that should be a separate design/performance change if we decide to guarantee deep isolation.

[namrataghadi-galileo]

Summary

• Adds AgentControl support for Orbit preset SLM scorer invocation and is paired with rungalileo/orbit#1720.
• Requires scorer_id; scorer_label is optional display metadata and scorer_version_id is a deprecated compatibility identifier.
• Calls /api/v1/scorers/invoke with a scoped internal HS256 JWT (scorers.invoke).
• Dual-writes legacy inputs and a structured record during the rollout, including selector-aligned input/output plus Step context, tools, and ground truth.
• Sends an Orbit execution timeout below the AgentControl HTTP deadline (8 seconds for the default 10-second deadline).

Scope

• Orbit owns feature-flag routing between RuntimeScorer and the legacy adapter.
• Nine flat preset SLM scorers are validated for direct runtime execution.
• Existing Step context propagation through models, engine, integrations, and SDKs is retained.
• Broader preset SLM runtime coverage is deferred to Jira phases 2 and 3.
• Customer-owned Luna scorers are not supported in this release.
• No backend selector, Protect payload construction, Wizard configuration, trusted tenant identity, or Galileo record-tree construction is added.

Risk and Rollout

• Risk level: medium.
• Deploy Orbit #1720 and keep its legacy adapter available for the tandem rollout.
• Roll back AgentControl to the legacy inputs.query/inputs.response contract if needed.
• Retain dual-write until older supported Orbit versions are retired.

Testing

• Galileo evaluator tests (116 passed)
• Models tests (71 passed)
• Engine tests (117 passed)
• Affected Python SDK tests (103 passed)
• Built-in evaluator tests (310 passed)
• TypeScript SDK tests, lint, and typecheck
• Repository lint, mypy, and git diff --check
• Full Python SDK target requires local PostgreSQL on localhost:5432

Follow-up

Orbit #1720 must update all six AgentControl git references to commit 8a2b5d52717090ed7c65b627035f73d89c46f5ff and regenerate requirements before the tandem merge.

[codecov]

Codecov Report

❌ Patch coverage is 99.25373% with 1 line in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...src/agent_control_evaluator_galileo/luna/client.py	96.96%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[abhinav-galileo]

[P1][pre-existing] Make cached async clients event-loop-safe

Cached evaluator instances retain pooled httpx.AsyncClient objects, while the synchronous decorator creates a new event loop per call. Reusing a keep-alive connection on a later loop can fail with RuntimeError: Event loop is closed, and cache eviction does not close these clients. This predates this PR and is not a regression introduced here; please track loop-scoped ownership or a persistent-loop lifecycle with awaited cleanup separately.

[namrataghadi-galileo]

@abhinav-galileo Review follow-up for 8a2b5d5:

• Structured record: addressed using Orbit #1720 as the authoritative contract. Orbit rejects conflicts between legacy inputs.query/inputs.response and record.input/record.output, so selector-selected values are intentionally dual-written. The full Step still supplies context, tools, dataset output, type, and name; the contract test verifies those structured fields are preserved.
• Scorer version identity: addressed. Metadata now uses requested_scorer_version_id, and docs state that Orbit executes the scorer's current default version.
• Timeout ordering: addressed. The server timeout is 80% of the HTTP deadline, and equal/greater explicit values fail locally before any request. Tests cover all branches.

The dependency-floor, invocation-resolved ADK tools, forward-compatible config, and deep Step snapshot threads are not addressed in this scoped change and remain open. Config strictness currently mirrors Orbit #1720's extra="forbid" contract.

[namrataghadi-galileo]

@abhinav-galileo Follow-up on the four remaining review threads:

• Dependency floors (#discussion_r3865624436): addressed in bdd30d5. The coordinated 8.6.0 floors now cover engine models/evaluators, builtin models and the Galileo extra, Galileo models/evaluators, and SDK/server runtime evaluators. evaluate_with_context(data, step) remains the additive hook, so evaluators implementing only evaluate(data) stay compatible.
• Invocation-resolved ADK tools (#discussion_r3865624464): addressed in bdd30d5. Model callbacks now prefer the exact current LlmRequest.tools_dict, treat an empty resolved map as authoritative, fall back to the bind-time registry only for older ADK versions, and reuse the captured set for the matching post-model evaluation. Tests cover dynamic override and an empty invocation set.
• Forward-compatible scorer config (#discussion_r3865624472): no code change is needed for this rollout. Orbit #1720 is authoritative and defines extra="forbid" with only threshold, score_threshold, and request_timeout_seconds. AgentControl intentionally mirrors that allowlist and should extend it in lockstep with Orbit.
• Deep Step snapshots (#discussion_r3865624478): no code change is needed for the scorer-invoke rollout. Luna only reads the trusted request-scoped Step. True evaluator isolation would require copying both selector-selected data and the full Step per evaluator in the hot path, so that should be a separate design/performance change if we decide to guarantee deep isolation.

[namrataghadi-galileo]

@abhinav-galileo Follow-up on the four remaining review threads:

• Dependency floors (#discussion_r3865624436): addressed in bdd30d5. The coordinated 8.6.0 floors now cover engine models/evaluators, builtin models and the Galileo extra, Galileo models/evaluators, and SDK/server runtime evaluators. evaluate_with_context(data, step) remains the additive hook, so evaluators implementing only evaluate(data) stay compatible.
• Invocation-resolved ADK tools (#discussion_r3865624464): addressed in bdd30d5. Model callbacks now prefer the exact current LlmRequest.tools_dict, treat an empty resolved map as authoritative, fall back to the bind-time registry only for older ADK versions, and reuse the captured set for the matching post-model evaluation. Tests cover dynamic override and an empty invocation set.
• Forward-compatible scorer config (#discussion_r3865624472): no code change is needed for this rollout. Orbit #1720 is authoritative and defines extra="forbid" with only threshold, score_threshold, and request_timeout_seconds. AgentControl intentionally mirrors that allowlist and should extend it in lockstep with Orbit.
• Deep Step snapshots (#discussion_r3865624478): no code change is needed for the scorer-invoke rollout. Luna only reads the trusted request-scoped Step. True evaluator isolation would require copying both selector-selected data and the full Step per evaluator in the hot path, so that should be a separate design/performance change if we decide to guarantee deep isolation.

[namrataghadi-galileo]

Summary

• Adds AgentControl support for Orbit preset SLM scorer invocation and is paired with rungalileo/orbit#1720.
• Requires scorer_id; scorer_label is optional display metadata and scorer_version_id is a deprecated compatibility identifier.
• Calls /api/v1/scorers/invoke with a scoped internal HS256 JWT (scorers.invoke).
• Dual-writes legacy inputs and a structured record during the rollout, including selector-aligned input/output plus Step context, tools, and ground truth.
• Sends an Orbit execution timeout below the AgentControl HTTP deadline (8 seconds for the default 10-second deadline).

Scope

• Orbit owns feature-flag routing between RuntimeScorer and the legacy adapter.
• Nine flat preset SLM scorers are validated for direct runtime execution.
• Existing Step context propagation through models, engine, integrations, and SDKs is retained.
• Broader preset SLM runtime coverage is deferred to Jira phases 2 and 3.
• Customer-owned Luna scorers are not supported in this release.
• No backend selector, Protect payload construction, Wizard configuration, trusted tenant identity, or Galileo record-tree construction is added.

Risk and Rollout

• Risk level: medium.
• Deploy Orbit #1720 and keep its legacy adapter available for the tandem rollout.
• Roll back AgentControl to the legacy inputs.query/inputs.response contract if needed.
• Retain dual-write until older supported Orbit versions are retired.

Testing

• Galileo evaluator tests (116 passed)
• Models tests (71 passed)
• Engine tests (117 passed)
• Affected Python SDK tests (103 passed)
• Built-in evaluator tests (310 passed)
• TypeScript SDK tests, lint, and typecheck
• Repository lint, mypy, and git diff --check
• Full Python SDK target requires local PostgreSQL on localhost:5432

Follow-up

Orbit #1720 must update all six AgentControl git references to commit 8a2b5d52717090ed7c65b627035f73d89c46f5ff and regenerate requirements before the tandem merge.

[codecov]

Codecov Report

❌ Patch coverage is 99.25373% with 1 line in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...src/agent_control_evaluator_galileo/luna/client.py	96.96%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[abhinav-galileo]

[P1][pre-existing] Make cached async clients event-loop-safe

Cached evaluator instances retain pooled httpx.AsyncClient objects, while the synchronous decorator creates a new event loop per call. Reusing a keep-alive connection on a later loop can fail with RuntimeError: Event loop is closed, and cache eviction does not close these clients. This predates this PR and is not a regression introduced here; please track loop-scoped ownership or a persistent-loop lifecycle with awaited cleanup separately.

[namrataghadi-galileo]

@abhinav-galileo Review follow-up for 8a2b5d5:

• Structured record: addressed using Orbit #1720 as the authoritative contract. Orbit rejects conflicts between legacy inputs.query/inputs.response and record.input/record.output, so selector-selected values are intentionally dual-written. The full Step still supplies context, tools, dataset output, type, and name; the contract test verifies those structured fields are preserved.
• Scorer version identity: addressed. Metadata now uses requested_scorer_version_id, and docs state that Orbit executes the scorer's current default version.
• Timeout ordering: addressed. The server timeout is 80% of the HTTP deadline, and equal/greater explicit values fail locally before any request. Tests cover all branches.

The dependency-floor, invocation-resolved ADK tools, forward-compatible config, and deep Step snapshot threads are not addressed in this scoped change and remain open. Config strictness currently mirrors Orbit #1720's extra="forbid" contract.

[namrataghadi-galileo]

@abhinav-galileo Follow-up on the four remaining review threads:

• Dependency floors (#discussion_r3865624436): addressed in bdd30d5. The coordinated 8.6.0 floors now cover engine models/evaluators, builtin models and the Galileo extra, Galileo models/evaluators, and SDK/server runtime evaluators. evaluate_with_context(data, step) remains the additive hook, so evaluators implementing only evaluate(data) stay compatible.
• Invocation-resolved ADK tools (#discussion_r3865624464): addressed in bdd30d5. Model callbacks now prefer the exact current LlmRequest.tools_dict, treat an empty resolved map as authoritative, fall back to the bind-time registry only for older ADK versions, and reuse the captured set for the matching post-model evaluation. Tests cover dynamic override and an empty invocation set.
• Forward-compatible scorer config (#discussion_r3865624472): no code change is needed for this rollout. Orbit #1720 is authoritative and defines extra="forbid" with only threshold, score_threshold, and request_timeout_seconds. AgentControl intentionally mirrors that allowlist and should extend it in lockstep with Orbit.
• Deep Step snapshots (#discussion_r3865624478): no code change is needed for the scorer-invoke rollout. Luna only reads the trusted request-scoped Step. True evaluator isolation would require copying both selector-selected data and the full Step per evaluator in the hot path, so that should be a separate design/performance change if we decide to guarantee deep isolation.

[namrataghadi-galileo]

@abhinav-galileo Follow-up on the four remaining review threads:

• Dependency floors (#discussion_r3865624436): addressed in bdd30d5. The coordinated 8.6.0 floors now cover engine models/evaluators, builtin models and the Galileo extra, Galileo models/evaluators, and SDK/server runtime evaluators. evaluate_with_context(data, step) remains the additive hook, so evaluators implementing only evaluate(data) stay compatible.
• Invocation-resolved ADK tools (#discussion_r3865624464): addressed in bdd30d5. Model callbacks now prefer the exact current LlmRequest.tools_dict, treat an empty resolved map as authoritative, fall back to the bind-time registry only for older ADK versions, and reuse the captured set for the matching post-model evaluation. Tests cover dynamic override and an empty invocation set.
• Forward-compatible scorer config (#discussion_r3865624472): no code change is needed for this rollout. Orbit #1720 is authoritative and defines extra="forbid" with only threshold, score_threshold, and request_timeout_seconds. AgentControl intentionally mirrors that allowlist and should extend it in lockstep with Orbit.
• Deep Step snapshots (#discussion_r3865624478): no code change is needed for the scorer-invoke rollout. Luna only reads the trusted request-scoped Step. True evaluator isolation would require copying both selector-selected data and the full Step per evaluator in the hot path, so that should be a separate design/performance change if we decide to guarantee deep isolation.

[namrataghadi-galileo]

Summary

• Adds AgentControl support for Orbit preset SLM scorer invocation and is paired with rungalileo/orbit#1720.
• Requires scorer_id; scorer_label is optional display metadata and scorer_version_id is a deprecated compatibility identifier.
• Calls /api/v1/scorers/invoke with a scoped internal HS256 JWT (scorers.invoke).
• Dual-writes legacy inputs and a structured record during the rollout, including selector-aligned input/output plus Step context, tools, and ground truth.
• Sends an Orbit execution timeout below the AgentControl HTTP deadline (8 seconds for the default 10-second deadline).

Scope

• Orbit owns feature-flag routing between RuntimeScorer and the legacy adapter.
• Nine flat preset SLM scorers are validated for direct runtime execution.
• Existing Step context propagation through models, engine, integrations, and SDKs is retained.
• Broader preset SLM runtime coverage is deferred to Jira phases 2 and 3.
• Customer-owned Luna scorers are not supported in this release.
• No backend selector, Protect payload construction, Wizard configuration, trusted tenant identity, or Galileo record-tree construction is added.

Risk and Rollout

• Risk level: medium.
• Deploy Orbit #1720 and keep its legacy adapter available for the tandem rollout.
• Roll back AgentControl to the legacy inputs.query/inputs.response contract if needed.
• Retain dual-write until older supported Orbit versions are retired.

Testing

• Galileo evaluator tests (116 passed)
• Models tests (71 passed)
• Engine tests (117 passed)
• Affected Python SDK tests (103 passed)
• Built-in evaluator tests (310 passed)
• TypeScript SDK tests, lint, and typecheck
• Repository lint, mypy, and git diff --check
• Full Python SDK target requires local PostgreSQL on localhost:5432

Follow-up

Orbit #1720 must update all six AgentControl git references to commit 8a2b5d52717090ed7c65b627035f73d89c46f5ff and regenerate requirements before the tandem merge.

[codecov]

Codecov Report

❌ Patch coverage is 99.25373% with 1 line in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...src/agent_control_evaluator_galileo/luna/client.py	96.96%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[abhinav-galileo]

[P1][pre-existing] Make cached async clients event-loop-safe

Cached evaluator instances retain pooled httpx.AsyncClient objects, while the synchronous decorator creates a new event loop per call. Reusing a keep-alive connection on a later loop can fail with RuntimeError: Event loop is closed, and cache eviction does not close these clients. This predates this PR and is not a regression introduced here; please track loop-scoped ownership or a persistent-loop lifecycle with awaited cleanup separately.

[namrataghadi-galileo]

@abhinav-galileo Review follow-up for 8a2b5d5:

• Structured record: addressed using Orbit #1720 as the authoritative contract. Orbit rejects conflicts between legacy inputs.query/inputs.response and record.input/record.output, so selector-selected values are intentionally dual-written. The full Step still supplies context, tools, dataset output, type, and name; the contract test verifies those structured fields are preserved.
• Scorer version identity: addressed. Metadata now uses requested_scorer_version_id, and docs state that Orbit executes the scorer's current default version.
• Timeout ordering: addressed. The server timeout is 80% of the HTTP deadline, and equal/greater explicit values fail locally before any request. Tests cover all branches.

The dependency-floor, invocation-resolved ADK tools, forward-compatible config, and deep Step snapshot threads are not addressed in this scoped change and remain open. Config strictness currently mirrors Orbit #1720's extra="forbid" contract.

[namrataghadi-galileo]

@abhinav-galileo Follow-up on the four remaining review threads:

• Dependency floors (#discussion_r3865624436): addressed in bdd30d5. The coordinated 8.6.0 floors now cover engine models/evaluators, builtin models and the Galileo extra, Galileo models/evaluators, and SDK/server runtime evaluators. evaluate_with_context(data, step) remains the additive hook, so evaluators implementing only evaluate(data) stay compatible.
• Invocation-resolved ADK tools (#discussion_r3865624464): addressed in bdd30d5. Model callbacks now prefer the exact current LlmRequest.tools_dict, treat an empty resolved map as authoritative, fall back to the bind-time registry only for older ADK versions, and reuse the captured set for the matching post-model evaluation. Tests cover dynamic override and an empty invocation set.
• Forward-compatible scorer config (#discussion_r3865624472): no code change is needed for this rollout. Orbit #1720 is authoritative and defines extra="forbid" with only threshold, score_threshold, and request_timeout_seconds. AgentControl intentionally mirrors that allowlist and should extend it in lockstep with Orbit.
• Deep Step snapshots (#discussion_r3865624478): no code change is needed for the scorer-invoke rollout. Luna only reads the trusted request-scoped Step. True evaluator isolation would require copying both selector-selected data and the full Step per evaluator in the hot path, so that should be a separate design/performance change if we decide to guarantee deep isolation.

[namrataghadi-galileo]

@abhinav-galileo Follow-up on the four remaining review threads:

• Dependency floors (#discussion_r3865624436): addressed in bdd30d5. The coordinated 8.6.0 floors now cover engine models/evaluators, builtin models and the Galileo extra, Galileo models/evaluators, and SDK/server runtime evaluators. evaluate_with_context(data, step) remains the additive hook, so evaluators implementing only evaluate(data) stay compatible.
• Invocation-resolved ADK tools (#discussion_r3865624464): addressed in bdd30d5. Model callbacks now prefer the exact current LlmRequest.tools_dict, treat an empty resolved map as authoritative, fall back to the bind-time registry only for older ADK versions, and reuse the captured set for the matching post-model evaluation. Tests cover dynamic override and an empty invocation set.
• Forward-compatible scorer config (#discussion_r3865624472): no code change is needed for this rollout. Orbit #1720 is authoritative and defines extra="forbid" with only threshold, score_threshold, and request_timeout_seconds. AgentControl intentionally mirrors that allowlist and should extend it in lockstep with Orbit.
• Deep Step snapshots (#discussion_r3865624478): no code change is needed for the scorer-invoke rollout. Luna only reads the trusted request-scoped Step. True evaluator isolation would require copying both selector-selected data and the full Step per evaluator in the hot path, so that should be a separate design/performance change if we decide to guarantee deep isolation.