refactor(evaluators)!: reorganize into builtin + extra tiers

[abhinav-galileo]

Summary

Split evaluators into a two-tier package architecture:

• builtin (agent-control-evaluators): Core infrastructure + regex, list, json, sql evaluators
• extra/galileo (agent-control-evaluator-galileo): Luna2 evaluator (calls external Galileo API)

Key Changes

Package Structure

• Move builtin evaluators to evaluators/builtin/
• Create evaluators/extra/galileo/ as a separate package
• Add entry points for plugin discovery (agent_control.evaluators)
• Update workspace to include only builtin (extras excluded for performance)
• Add CI workflow for testing extra packages (.github/workflows/test-extras.yml)
• Add template scaffold for creating new evaluator packages (evaluators/extra/template/)

Dependency Strategy

• SDK and server depend on agent-control-evaluators as a runtime dependency (not vendored)
• This avoids duplicate module conflicts when galileo extras are installed
• Server build script no longer vendors evaluators (only models + engine)

Naming Convention

• External evaluator names now use dot notation instead of slash
• Example: galileo.luna2 instead of galileo/luna2

Breaking Changes

1. Python Imports

# OLD (will break):fromagent_control_evaluators.galileo_luna2importLuna2Evaluator# NEW:fromagent_control_evaluator_galileo.luna2importLuna2Evaluator# OR via SDK (still works):fromagent_control.evaluatorsimportLuna2Evaluator

2. Evaluator Names

# OLD:evaluator_name="galileo/luna2"# NEW:evaluator_name="galileo.luna2"

3. Installation

# Builtin evaluators (automatic with SDK/server):
pip install agent-control-evaluators
# Galileo Luna2 (optional):
pip install agent-control-evaluator-galileo
# Or via convenience extra:
pip install agent-control-evaluators[galileo]

Migration Guide

1. Install the new package: pip install agent-control-evaluator-galileo
2. Update Python imports from agent_control_evaluators.galileo_luna2 to agent_control_evaluator_galileo.luna2
3. Database migration required: Update controls where data.evaluator.name contains / to use .:

   UPDATE controls
   SET data = jsonb_set(data, '{evaluator,name}',
   to_jsonb(replace(data->'evaluator'->>'name', '/', '.')))
   WHERE data->'evaluator'->>'name'LIKE'%/%'AND data->'evaluator'->>'name' NOT LIKE'%:%';

Test Plan

• All existing tests pass (771 total)
• Lint passes for all packages
• Type check passes for all packages
• Luna2 smoke test correctly skips when package not installed
• Entry points register builtin evaluators automatically

[codecov]

The author of this PR, abhinav-galileo, is not an activated member of this organization on Codecov.
Please activate this user on Codecov to display this PR comment.
Coverage data is still being uploaded to Codecov.io for purposes of overall coverage calculations.
Please don't hesitate to email us at support@codecov.io with any questions.

[lan17]

Posting review notes from my local pass:

1. JSON evaluator extra-field check can false-positive when allow_extra_fields=false because the allow-list only includes field_types and required_fields; fields referenced via field_constraints or field_patterns are treated as “extra”. File: evaluators/builtin/src/agent_control_evaluators/json/evaluator.py (extra_paths logic).

2. Docs/packaging mismatch: README suggests pip install agent-control-evaluators[galileo], but the [galileo] extra is commented out in evaluators/builtin/pyproject.toml, so that command currently fails. Suggest enabling the extra when published or pointing docs to agent-control-evaluator-galileo.

3. Extras tests not wired: make test only runs builtin evaluators; evaluators/extra/galileo/tests aren’t exercised by default (and I don’t see a test-extras workflow). Consider adding a target/workflow or documenting the omission.

[lan17]

To supportagent-control-evaluators[xyz]installs:

1. Enable extras on the builtin package

   • In evaluators/builtin/pyproject.toml, add entries under [project.optional-dependencies], e.g.
     • galileo = ["agent-control-evaluator-galileo>=3.0.0"]
     • nvidia = ["agent-control-evaluator-nvidia>=3.0.0"] (when published)

2. Publish provider packages

   • Make sure the release workflow publishes agent-control-evaluator-<provider> packages before or alongsideagent-control-evaluators, so the extras resolve on PyPI.

3. Optional (if you want singular syntax)

   • If you prefer agent-control-evaluator[galileo], add a small meta package named agent-control-evaluator that depends on agent-control-evaluators and defines the same extras.

Right now the extras are commented out, so agent-control-evaluators[galileo] will not resolve until #1 + #2 are done.

[abhinav-galileo]

Summary

Split evaluators into a two-tier package architecture:

• builtin (agent-control-evaluators): Core infrastructure + regex, list, json, sql evaluators
• extra/galileo (agent-control-evaluator-galileo): Luna2 evaluator (calls external Galileo API)

Key Changes

Package Structure

• Move builtin evaluators to evaluators/builtin/
• Create evaluators/extra/galileo/ as a separate package
• Add entry points for plugin discovery (agent_control.evaluators)
• Update workspace to include only builtin (extras excluded for performance)
• Add CI workflow for testing extra packages (.github/workflows/test-extras.yml)
• Add template scaffold for creating new evaluator packages (evaluators/extra/template/)

Dependency Strategy

• SDK and server depend on agent-control-evaluators as a runtime dependency (not vendored)
• This avoids duplicate module conflicts when galileo extras are installed
• Server build script no longer vendors evaluators (only models + engine)

Naming Convention

• External evaluator names now use dot notation instead of slash
• Example: galileo.luna2 instead of galileo/luna2

Breaking Changes

1. Python Imports

# OLD (will break):fromagent_control_evaluators.galileo_luna2importLuna2Evaluator# NEW:fromagent_control_evaluator_galileo.luna2importLuna2Evaluator# OR via SDK (still works):fromagent_control.evaluatorsimportLuna2Evaluator

2. Evaluator Names

# OLD:evaluator_name="galileo/luna2"# NEW:evaluator_name="galileo.luna2"

3. Installation

# Builtin evaluators (automatic with SDK/server):
pip install agent-control-evaluators
# Galileo Luna2 (optional):
pip install agent-control-evaluator-galileo
# Or via convenience extra:
pip install agent-control-evaluators[galileo]

Migration Guide

1. Install the new package: pip install agent-control-evaluator-galileo
2. Update Python imports from agent_control_evaluators.galileo_luna2 to agent_control_evaluator_galileo.luna2
3. Database migration required: Update controls where data.evaluator.name contains / to use .:

   UPDATE controls
   SET data = jsonb_set(data, '{evaluator,name}',
   to_jsonb(replace(data->'evaluator'->>'name', '/', '.')))
   WHERE data->'evaluator'->>'name'LIKE'%/%'AND data->'evaluator'->>'name' NOT LIKE'%:%';

Test Plan

• All existing tests pass (771 total)
• Lint passes for all packages
• Type check passes for all packages
• Luna2 smoke test correctly skips when package not installed
• Entry points register builtin evaluators automatically

[codecov]

The author of this PR, abhinav-galileo, is not an activated member of this organization on Codecov.
Please activate this user on Codecov to display this PR comment.
Coverage data is still being uploaded to Codecov.io for purposes of overall coverage calculations.
Please don't hesitate to email us at support@codecov.io with any questions.

[lan17]

Posting review notes from my local pass:

1. JSON evaluator extra-field check can false-positive when allow_extra_fields=false because the allow-list only includes field_types and required_fields; fields referenced via field_constraints or field_patterns are treated as “extra”. File: evaluators/builtin/src/agent_control_evaluators/json/evaluator.py (extra_paths logic).

2. Docs/packaging mismatch: README suggests pip install agent-control-evaluators[galileo], but the [galileo] extra is commented out in evaluators/builtin/pyproject.toml, so that command currently fails. Suggest enabling the extra when published or pointing docs to agent-control-evaluator-galileo.

3. Extras tests not wired: make test only runs builtin evaluators; evaluators/extra/galileo/tests aren’t exercised by default (and I don’t see a test-extras workflow). Consider adding a target/workflow or documenting the omission.

[lan17]

To supportagent-control-evaluators[xyz]installs:

1. Enable extras on the builtin package

   • In evaluators/builtin/pyproject.toml, add entries under [project.optional-dependencies], e.g.
     • galileo = ["agent-control-evaluator-galileo>=3.0.0"]
     • nvidia = ["agent-control-evaluator-nvidia>=3.0.0"] (when published)

2. Publish provider packages

   • Make sure the release workflow publishes agent-control-evaluator-<provider> packages before or alongsideagent-control-evaluators, so the extras resolve on PyPI.

3. Optional (if you want singular syntax)

   • If you prefer agent-control-evaluator[galileo], add a small meta package named agent-control-evaluator that depends on agent-control-evaluators and defines the same extras.

Right now the extras are commented out, so agent-control-evaluators[galileo] will not resolve until #1 + #2 are done.

[abhinav-galileo]

Summary

Split evaluators into a two-tier package architecture:

• builtin (agent-control-evaluators): Core infrastructure + regex, list, json, sql evaluators
• extra/galileo (agent-control-evaluator-galileo): Luna2 evaluator (calls external Galileo API)

Key Changes

Package Structure

• Move builtin evaluators to evaluators/builtin/
• Create evaluators/extra/galileo/ as a separate package
• Add entry points for plugin discovery (agent_control.evaluators)
• Update workspace to include only builtin (extras excluded for performance)
• Add CI workflow for testing extra packages (.github/workflows/test-extras.yml)
• Add template scaffold for creating new evaluator packages (evaluators/extra/template/)

Dependency Strategy

• SDK and server depend on agent-control-evaluators as a runtime dependency (not vendored)
• This avoids duplicate module conflicts when galileo extras are installed
• Server build script no longer vendors evaluators (only models + engine)

Naming Convention

• External evaluator names now use dot notation instead of slash
• Example: galileo.luna2 instead of galileo/luna2

Breaking Changes

1. Python Imports

# OLD (will break):fromagent_control_evaluators.galileo_luna2importLuna2Evaluator# NEW:fromagent_control_evaluator_galileo.luna2importLuna2Evaluator# OR via SDK (still works):fromagent_control.evaluatorsimportLuna2Evaluator

2. Evaluator Names

# OLD:evaluator_name="galileo/luna2"# NEW:evaluator_name="galileo.luna2"

3. Installation

# Builtin evaluators (automatic with SDK/server):
pip install agent-control-evaluators
# Galileo Luna2 (optional):
pip install agent-control-evaluator-galileo
# Or via convenience extra:
pip install agent-control-evaluators[galileo]

Migration Guide

1. Install the new package: pip install agent-control-evaluator-galileo
2. Update Python imports from agent_control_evaluators.galileo_luna2 to agent_control_evaluator_galileo.luna2
3. Database migration required: Update controls where data.evaluator.name contains / to use .:

   UPDATE controls
   SET data = jsonb_set(data, '{evaluator,name}',
   to_jsonb(replace(data->'evaluator'->>'name', '/', '.')))
   WHERE data->'evaluator'->>'name'LIKE'%/%'AND data->'evaluator'->>'name' NOT LIKE'%:%';

Test Plan

• All existing tests pass (771 total)
• Lint passes for all packages
• Type check passes for all packages
• Luna2 smoke test correctly skips when package not installed
• Entry points register builtin evaluators automatically

[codecov]

The author of this PR, abhinav-galileo, is not an activated member of this organization on Codecov.
Please activate this user on Codecov to display this PR comment.
Coverage data is still being uploaded to Codecov.io for purposes of overall coverage calculations.
Please don't hesitate to email us at support@codecov.io with any questions.

[lan17]

Posting review notes from my local pass:

1. JSON evaluator extra-field check can false-positive when allow_extra_fields=false because the allow-list only includes field_types and required_fields; fields referenced via field_constraints or field_patterns are treated as “extra”. File: evaluators/builtin/src/agent_control_evaluators/json/evaluator.py (extra_paths logic).

2. Docs/packaging mismatch: README suggests pip install agent-control-evaluators[galileo], but the [galileo] extra is commented out in evaluators/builtin/pyproject.toml, so that command currently fails. Suggest enabling the extra when published or pointing docs to agent-control-evaluator-galileo.

3. Extras tests not wired: make test only runs builtin evaluators; evaluators/extra/galileo/tests aren’t exercised by default (and I don’t see a test-extras workflow). Consider adding a target/workflow or documenting the omission.

[lan17]

To supportagent-control-evaluators[xyz]installs:

1. Enable extras on the builtin package

   • In evaluators/builtin/pyproject.toml, add entries under [project.optional-dependencies], e.g.
     • galileo = ["agent-control-evaluator-galileo>=3.0.0"]
     • nvidia = ["agent-control-evaluator-nvidia>=3.0.0"] (when published)

2. Publish provider packages

   • Make sure the release workflow publishes agent-control-evaluator-<provider> packages before or alongsideagent-control-evaluators, so the extras resolve on PyPI.

3. Optional (if you want singular syntax)

   • If you prefer agent-control-evaluator[galileo], add a small meta package named agent-control-evaluator that depends on agent-control-evaluators and defines the same extras.

Right now the extras are commented out, so agent-control-evaluators[galileo] will not resolve until #1 + #2 are done.

[abhinav-galileo]

Summary

Split evaluators into a two-tier package architecture:

• builtin (agent-control-evaluators): Core infrastructure + regex, list, json, sql evaluators
• extra/galileo (agent-control-evaluator-galileo): Luna2 evaluator (calls external Galileo API)

Key Changes

Package Structure

• Move builtin evaluators to evaluators/builtin/
• Create evaluators/extra/galileo/ as a separate package
• Add entry points for plugin discovery (agent_control.evaluators)
• Update workspace to include only builtin (extras excluded for performance)
• Add CI workflow for testing extra packages (.github/workflows/test-extras.yml)
• Add template scaffold for creating new evaluator packages (evaluators/extra/template/)

Dependency Strategy

• SDK and server depend on agent-control-evaluators as a runtime dependency (not vendored)
• This avoids duplicate module conflicts when galileo extras are installed
• Server build script no longer vendors evaluators (only models + engine)

Naming Convention

• External evaluator names now use dot notation instead of slash
• Example: galileo.luna2 instead of galileo/luna2

Breaking Changes

1. Python Imports

# OLD (will break):fromagent_control_evaluators.galileo_luna2importLuna2Evaluator# NEW:fromagent_control_evaluator_galileo.luna2importLuna2Evaluator# OR via SDK (still works):fromagent_control.evaluatorsimportLuna2Evaluator

2. Evaluator Names

# OLD:evaluator_name="galileo/luna2"# NEW:evaluator_name="galileo.luna2"

3. Installation

# Builtin evaluators (automatic with SDK/server):
pip install agent-control-evaluators
# Galileo Luna2 (optional):
pip install agent-control-evaluator-galileo
# Or via convenience extra:
pip install agent-control-evaluators[galileo]

Migration Guide

1. Install the new package: pip install agent-control-evaluator-galileo
2. Update Python imports from agent_control_evaluators.galileo_luna2 to agent_control_evaluator_galileo.luna2
3. Database migration required: Update controls where data.evaluator.name contains / to use .:

   UPDATE controls
   SET data = jsonb_set(data, '{evaluator,name}',
   to_jsonb(replace(data->'evaluator'->>'name', '/', '.')))
   WHERE data->'evaluator'->>'name'LIKE'%/%'AND data->'evaluator'->>'name' NOT LIKE'%:%';

Test Plan

• All existing tests pass (771 total)
• Lint passes for all packages
• Type check passes for all packages
• Luna2 smoke test correctly skips when package not installed
• Entry points register builtin evaluators automatically

[codecov]

The author of this PR, abhinav-galileo, is not an activated member of this organization on Codecov.
Please activate this user on Codecov to display this PR comment.
Coverage data is still being uploaded to Codecov.io for purposes of overall coverage calculations.
Please don't hesitate to email us at support@codecov.io with any questions.

[lan17]

Posting review notes from my local pass:

1. JSON evaluator extra-field check can false-positive when allow_extra_fields=false because the allow-list only includes field_types and required_fields; fields referenced via field_constraints or field_patterns are treated as “extra”. File: evaluators/builtin/src/agent_control_evaluators/json/evaluator.py (extra_paths logic).

2. Docs/packaging mismatch: README suggests pip install agent-control-evaluators[galileo], but the [galileo] extra is commented out in evaluators/builtin/pyproject.toml, so that command currently fails. Suggest enabling the extra when published or pointing docs to agent-control-evaluator-galileo.

3. Extras tests not wired: make test only runs builtin evaluators; evaluators/extra/galileo/tests aren’t exercised by default (and I don’t see a test-extras workflow). Consider adding a target/workflow or documenting the omission.

[lan17]

To supportagent-control-evaluators[xyz]installs:

1. Enable extras on the builtin package

   • In evaluators/builtin/pyproject.toml, add entries under [project.optional-dependencies], e.g.
     • galileo = ["agent-control-evaluator-galileo>=3.0.0"]
     • nvidia = ["agent-control-evaluator-nvidia>=3.0.0"] (when published)

2. Publish provider packages

   • Make sure the release workflow publishes agent-control-evaluator-<provider> packages before or alongsideagent-control-evaluators, so the extras resolve on PyPI.

3. Optional (if you want singular syntax)

   • If you prefer agent-control-evaluator[galileo], add a small meta package named agent-control-evaluator that depends on agent-control-evaluators and defines the same extras.

Right now the extras are commented out, so agent-control-evaluators[galileo] will not resolve until #1 + #2 are done.

[abhinav-galileo]

Summary

Split evaluators into a two-tier package architecture:

• builtin (agent-control-evaluators): Core infrastructure + regex, list, json, sql evaluators
• extra/galileo (agent-control-evaluator-galileo): Luna2 evaluator (calls external Galileo API)

Key Changes

Package Structure

• Move builtin evaluators to evaluators/builtin/
• Create evaluators/extra/galileo/ as a separate package
• Add entry points for plugin discovery (agent_control.evaluators)
• Update workspace to include only builtin (extras excluded for performance)
• Add CI workflow for testing extra packages (.github/workflows/test-extras.yml)
• Add template scaffold for creating new evaluator packages (evaluators/extra/template/)

Dependency Strategy

• SDK and server depend on agent-control-evaluators as a runtime dependency (not vendored)
• This avoids duplicate module conflicts when galileo extras are installed
• Server build script no longer vendors evaluators (only models + engine)

Naming Convention

• External evaluator names now use dot notation instead of slash
• Example: galileo.luna2 instead of galileo/luna2

Breaking Changes

1. Python Imports

# OLD (will break):fromagent_control_evaluators.galileo_luna2importLuna2Evaluator# NEW:fromagent_control_evaluator_galileo.luna2importLuna2Evaluator# OR via SDK (still works):fromagent_control.evaluatorsimportLuna2Evaluator

2. Evaluator Names

# OLD:evaluator_name="galileo/luna2"# NEW:evaluator_name="galileo.luna2"

3. Installation

# Builtin evaluators (automatic with SDK/server):
pip install agent-control-evaluators
# Galileo Luna2 (optional):
pip install agent-control-evaluator-galileo
# Or via convenience extra:
pip install agent-control-evaluators[galileo]

Migration Guide

1. Install the new package: pip install agent-control-evaluator-galileo
2. Update Python imports from agent_control_evaluators.galileo_luna2 to agent_control_evaluator_galileo.luna2
3. Database migration required: Update controls where data.evaluator.name contains / to use .:

   UPDATE controls
   SET data = jsonb_set(data, '{evaluator,name}',
   to_jsonb(replace(data->'evaluator'->>'name', '/', '.')))
   WHERE data->'evaluator'->>'name'LIKE'%/%'AND data->'evaluator'->>'name' NOT LIKE'%:%';

Test Plan

• All existing tests pass (771 total)
• Lint passes for all packages
• Type check passes for all packages
• Luna2 smoke test correctly skips when package not installed
• Entry points register builtin evaluators automatically

[codecov]

The author of this PR, abhinav-galileo, is not an activated member of this organization on Codecov.
Please activate this user on Codecov to display this PR comment.
Coverage data is still being uploaded to Codecov.io for purposes of overall coverage calculations.
Please don't hesitate to email us at support@codecov.io with any questions.

[lan17]

Posting review notes from my local pass:

1. JSON evaluator extra-field check can false-positive when allow_extra_fields=false because the allow-list only includes field_types and required_fields; fields referenced via field_constraints or field_patterns are treated as “extra”. File: evaluators/builtin/src/agent_control_evaluators/json/evaluator.py (extra_paths logic).

2. Docs/packaging mismatch: README suggests pip install agent-control-evaluators[galileo], but the [galileo] extra is commented out in evaluators/builtin/pyproject.toml, so that command currently fails. Suggest enabling the extra when published or pointing docs to agent-control-evaluator-galileo.

3. Extras tests not wired: make test only runs builtin evaluators; evaluators/extra/galileo/tests aren’t exercised by default (and I don’t see a test-extras workflow). Consider adding a target/workflow or documenting the omission.

[lan17]

To supportagent-control-evaluators[xyz]installs:

1. Enable extras on the builtin package

   • In evaluators/builtin/pyproject.toml, add entries under [project.optional-dependencies], e.g.
     • galileo = ["agent-control-evaluator-galileo>=3.0.0"]
     • nvidia = ["agent-control-evaluator-nvidia>=3.0.0"] (when published)

2. Publish provider packages

   • Make sure the release workflow publishes agent-control-evaluator-<provider> packages before or alongsideagent-control-evaluators, so the extras resolve on PyPI.

3. Optional (if you want singular syntax)

   • If you prefer agent-control-evaluator[galileo], add a small meta package named agent-control-evaluator that depends on agent-control-evaluators and defines the same extras.

Right now the extras are commented out, so agent-control-evaluators[galileo] will not resolve until #1 + #2 are done.

[abhinav-galileo]

Summary

Split evaluators into a two-tier package architecture:

• builtin (agent-control-evaluators): Core infrastructure + regex, list, json, sql evaluators
• extra/galileo (agent-control-evaluator-galileo): Luna2 evaluator (calls external Galileo API)

Key Changes

Package Structure

• Move builtin evaluators to evaluators/builtin/
• Create evaluators/extra/galileo/ as a separate package
• Add entry points for plugin discovery (agent_control.evaluators)
• Update workspace to include only builtin (extras excluded for performance)
• Add CI workflow for testing extra packages (.github/workflows/test-extras.yml)
• Add template scaffold for creating new evaluator packages (evaluators/extra/template/)

Dependency Strategy

• SDK and server depend on agent-control-evaluators as a runtime dependency (not vendored)
• This avoids duplicate module conflicts when galileo extras are installed
• Server build script no longer vendors evaluators (only models + engine)

Naming Convention

• External evaluator names now use dot notation instead of slash
• Example: galileo.luna2 instead of galileo/luna2

Breaking Changes

1. Python Imports

# OLD (will break):fromagent_control_evaluators.galileo_luna2importLuna2Evaluator# NEW:fromagent_control_evaluator_galileo.luna2importLuna2Evaluator# OR via SDK (still works):fromagent_control.evaluatorsimportLuna2Evaluator

2. Evaluator Names

# OLD:evaluator_name="galileo/luna2"# NEW:evaluator_name="galileo.luna2"

3. Installation

# Builtin evaluators (automatic with SDK/server):
pip install agent-control-evaluators
# Galileo Luna2 (optional):
pip install agent-control-evaluator-galileo
# Or via convenience extra:
pip install agent-control-evaluators[galileo]

Migration Guide

1. Install the new package: pip install agent-control-evaluator-galileo
2. Update Python imports from agent_control_evaluators.galileo_luna2 to agent_control_evaluator_galileo.luna2
3. Database migration required: Update controls where data.evaluator.name contains / to use .:

   UPDATE controls
   SET data = jsonb_set(data, '{evaluator,name}',
   to_jsonb(replace(data->'evaluator'->>'name', '/', '.')))
   WHERE data->'evaluator'->>'name'LIKE'%/%'AND data->'evaluator'->>'name' NOT LIKE'%:%';

Test Plan

• All existing tests pass (771 total)
• Lint passes for all packages
• Type check passes for all packages
• Luna2 smoke test correctly skips when package not installed
• Entry points register builtin evaluators automatically

[codecov]

The author of this PR, abhinav-galileo, is not an activated member of this organization on Codecov.
Please activate this user on Codecov to display this PR comment.
Coverage data is still being uploaded to Codecov.io for purposes of overall coverage calculations.
Please don't hesitate to email us at support@codecov.io with any questions.

[lan17]

Posting review notes from my local pass:

1. JSON evaluator extra-field check can false-positive when allow_extra_fields=false because the allow-list only includes field_types and required_fields; fields referenced via field_constraints or field_patterns are treated as “extra”. File: evaluators/builtin/src/agent_control_evaluators/json/evaluator.py (extra_paths logic).

2. Docs/packaging mismatch: README suggests pip install agent-control-evaluators[galileo], but the [galileo] extra is commented out in evaluators/builtin/pyproject.toml, so that command currently fails. Suggest enabling the extra when published or pointing docs to agent-control-evaluator-galileo.

3. Extras tests not wired: make test only runs builtin evaluators; evaluators/extra/galileo/tests aren’t exercised by default (and I don’t see a test-extras workflow). Consider adding a target/workflow or documenting the omission.

[lan17]

To supportagent-control-evaluators[xyz]installs:

1. Enable extras on the builtin package

   • In evaluators/builtin/pyproject.toml, add entries under [project.optional-dependencies], e.g.
     • galileo = ["agent-control-evaluator-galileo>=3.0.0"]
     • nvidia = ["agent-control-evaluator-nvidia>=3.0.0"] (when published)

2. Publish provider packages

   • Make sure the release workflow publishes agent-control-evaluator-<provider> packages before or alongsideagent-control-evaluators, so the extras resolve on PyPI.

3. Optional (if you want singular syntax)

   • If you prefer agent-control-evaluator[galileo], add a small meta package named agent-control-evaluator that depends on agent-control-evaluators and defines the same extras.

Right now the extras are commented out, so agent-control-evaluators[galileo] will not resolve until #1 + #2 are done.

[abhinav-galileo]

Summary

Split evaluators into a two-tier package architecture:

• builtin (agent-control-evaluators): Core infrastructure + regex, list, json, sql evaluators
• extra/galileo (agent-control-evaluator-galileo): Luna2 evaluator (calls external Galileo API)

Key Changes

Package Structure

• Move builtin evaluators to evaluators/builtin/
• Create evaluators/extra/galileo/ as a separate package
• Add entry points for plugin discovery (agent_control.evaluators)
• Update workspace to include only builtin (extras excluded for performance)
• Add CI workflow for testing extra packages (.github/workflows/test-extras.yml)
• Add template scaffold for creating new evaluator packages (evaluators/extra/template/)

Dependency Strategy

• SDK and server depend on agent-control-evaluators as a runtime dependency (not vendored)
• This avoids duplicate module conflicts when galileo extras are installed
• Server build script no longer vendors evaluators (only models + engine)

Naming Convention

• External evaluator names now use dot notation instead of slash
• Example: galileo.luna2 instead of galileo/luna2

Breaking Changes

1. Python Imports

# OLD (will break):fromagent_control_evaluators.galileo_luna2importLuna2Evaluator# NEW:fromagent_control_evaluator_galileo.luna2importLuna2Evaluator# OR via SDK (still works):fromagent_control.evaluatorsimportLuna2Evaluator

2. Evaluator Names

# OLD:evaluator_name="galileo/luna2"# NEW:evaluator_name="galileo.luna2"

3. Installation

# Builtin evaluators (automatic with SDK/server):
pip install agent-control-evaluators
# Galileo Luna2 (optional):
pip install agent-control-evaluator-galileo
# Or via convenience extra:
pip install agent-control-evaluators[galileo]

Migration Guide

1. Install the new package: pip install agent-control-evaluator-galileo
2. Update Python imports from agent_control_evaluators.galileo_luna2 to agent_control_evaluator_galileo.luna2
3. Database migration required: Update controls where data.evaluator.name contains / to use .:

   UPDATE controls
   SET data = jsonb_set(data, '{evaluator,name}',
   to_jsonb(replace(data->'evaluator'->>'name', '/', '.')))
   WHERE data->'evaluator'->>'name'LIKE'%/%'AND data->'evaluator'->>'name' NOT LIKE'%:%';

Test Plan

• All existing tests pass (771 total)
• Lint passes for all packages
• Type check passes for all packages
• Luna2 smoke test correctly skips when package not installed
• Entry points register builtin evaluators automatically

[codecov]

The author of this PR, abhinav-galileo, is not an activated member of this organization on Codecov.
Please activate this user on Codecov to display this PR comment.
Coverage data is still being uploaded to Codecov.io for purposes of overall coverage calculations.
Please don't hesitate to email us at support@codecov.io with any questions.

[lan17]

Posting review notes from my local pass:

1. JSON evaluator extra-field check can false-positive when allow_extra_fields=false because the allow-list only includes field_types and required_fields; fields referenced via field_constraints or field_patterns are treated as “extra”. File: evaluators/builtin/src/agent_control_evaluators/json/evaluator.py (extra_paths logic).

2. Docs/packaging mismatch: README suggests pip install agent-control-evaluators[galileo], but the [galileo] extra is commented out in evaluators/builtin/pyproject.toml, so that command currently fails. Suggest enabling the extra when published or pointing docs to agent-control-evaluator-galileo.

3. Extras tests not wired: make test only runs builtin evaluators; evaluators/extra/galileo/tests aren’t exercised by default (and I don’t see a test-extras workflow). Consider adding a target/workflow or documenting the omission.

[lan17]

To supportagent-control-evaluators[xyz]installs:

1. Enable extras on the builtin package

   • In evaluators/builtin/pyproject.toml, add entries under [project.optional-dependencies], e.g.
     • galileo = ["agent-control-evaluator-galileo>=3.0.0"]
     • nvidia = ["agent-control-evaluator-nvidia>=3.0.0"] (when published)

2. Publish provider packages

   • Make sure the release workflow publishes agent-control-evaluator-<provider> packages before or alongsideagent-control-evaluators, so the extras resolve on PyPI.

3. Optional (if you want singular syntax)

   • If you prefer agent-control-evaluator[galileo], add a small meta package named agent-control-evaluator that depends on agent-control-evaluators and defines the same extras.

Right now the extras are commented out, so agent-control-evaluators[galileo] will not resolve until #1 + #2 are done.

[abhinav-galileo]

Summary

Split evaluators into a two-tier package architecture:

• builtin (agent-control-evaluators): Core infrastructure + regex, list, json, sql evaluators
• extra/galileo (agent-control-evaluator-galileo): Luna2 evaluator (calls external Galileo API)

Key Changes

Package Structure

• Move builtin evaluators to evaluators/builtin/
• Create evaluators/extra/galileo/ as a separate package
• Add entry points for plugin discovery (agent_control.evaluators)
• Update workspace to include only builtin (extras excluded for performance)
• Add CI workflow for testing extra packages (.github/workflows/test-extras.yml)
• Add template scaffold for creating new evaluator packages (evaluators/extra/template/)

Dependency Strategy

• SDK and server depend on agent-control-evaluators as a runtime dependency (not vendored)
• This avoids duplicate module conflicts when galileo extras are installed
• Server build script no longer vendors evaluators (only models + engine)

Naming Convention

• External evaluator names now use dot notation instead of slash
• Example: galileo.luna2 instead of galileo/luna2

Breaking Changes

1. Python Imports

# OLD (will break):fromagent_control_evaluators.galileo_luna2importLuna2Evaluator# NEW:fromagent_control_evaluator_galileo.luna2importLuna2Evaluator# OR via SDK (still works):fromagent_control.evaluatorsimportLuna2Evaluator

2. Evaluator Names

# OLD:evaluator_name="galileo/luna2"# NEW:evaluator_name="galileo.luna2"

3. Installation

# Builtin evaluators (automatic with SDK/server):
pip install agent-control-evaluators
# Galileo Luna2 (optional):
pip install agent-control-evaluator-galileo
# Or via convenience extra:
pip install agent-control-evaluators[galileo]

Migration Guide

1. Install the new package: pip install agent-control-evaluator-galileo
2. Update Python imports from agent_control_evaluators.galileo_luna2 to agent_control_evaluator_galileo.luna2
3. Database migration required: Update controls where data.evaluator.name contains / to use .:

   UPDATE controls
   SET data = jsonb_set(data, '{evaluator,name}',
   to_jsonb(replace(data->'evaluator'->>'name', '/', '.')))
   WHERE data->'evaluator'->>'name'LIKE'%/%'AND data->'evaluator'->>'name' NOT LIKE'%:%';

Test Plan

• All existing tests pass (771 total)
• Lint passes for all packages
• Type check passes for all packages
• Luna2 smoke test correctly skips when package not installed
• Entry points register builtin evaluators automatically

[codecov]

The author of this PR, abhinav-galileo, is not an activated member of this organization on Codecov.
Please activate this user on Codecov to display this PR comment.
Coverage data is still being uploaded to Codecov.io for purposes of overall coverage calculations.
Please don't hesitate to email us at support@codecov.io with any questions.

[lan17]

Posting review notes from my local pass:

1. JSON evaluator extra-field check can false-positive when allow_extra_fields=false because the allow-list only includes field_types and required_fields; fields referenced via field_constraints or field_patterns are treated as “extra”. File: evaluators/builtin/src/agent_control_evaluators/json/evaluator.py (extra_paths logic).

2. Docs/packaging mismatch: README suggests pip install agent-control-evaluators[galileo], but the [galileo] extra is commented out in evaluators/builtin/pyproject.toml, so that command currently fails. Suggest enabling the extra when published or pointing docs to agent-control-evaluator-galileo.

3. Extras tests not wired: make test only runs builtin evaluators; evaluators/extra/galileo/tests aren’t exercised by default (and I don’t see a test-extras workflow). Consider adding a target/workflow or documenting the omission.

[lan17]

To supportagent-control-evaluators[xyz]installs:

1. Enable extras on the builtin package

   • In evaluators/builtin/pyproject.toml, add entries under [project.optional-dependencies], e.g.
     • galileo = ["agent-control-evaluator-galileo>=3.0.0"]
     • nvidia = ["agent-control-evaluator-nvidia>=3.0.0"] (when published)

2. Publish provider packages

   • Make sure the release workflow publishes agent-control-evaluator-<provider> packages before or alongsideagent-control-evaluators, so the extras resolve on PyPI.

3. Optional (if you want singular syntax)

   • If you prefer agent-control-evaluator[galileo], add a small meta package named agent-control-evaluator that depends on agent-control-evaluators and defines the same extras.

Right now the extras are commented out, so agent-control-evaluators[galileo] will not resolve until #1 + #2 are done.