Skip to content

feat(agents): add strict expert program frontend - #498

Merged
yuecideng merged 50 commits into
mainfrom
feat/mllm-expert-program-frontend
Aug 24, 2026
Merged

feat(agents): add strict expert program frontend#498
yuecideng merged 50 commits into
mainfrom
feat/mllm-expert-program-frontend

Conversation

@yuecideng

@yuecidengyuecideng commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Description

Stack

The frontend cannot invent unregistered calls, providers, scene references, or low-level controller commands, preserving one semantic facade from generation through execution.

Refs #471
Refs #474

Type of change

  • New feature (non-breaking change which adds functionality)

Screenshots

Not applicable.

Validation

  • Focused coverage: pytest -q tests/agents/mllm/test_expert_program.py — 21 passed
  • Final affected-suite regression on the stack tip: 1215 passed, 2 skipped, 8 deselected
  • Changed Python files pass Black 26.3.1; the Sphinx build and rollout-report drift check pass at the stack tip

Checklist

  • Changed Python files pass Black 26.3.1.
  • Corresponding public/design documentation is included in this stack.
  • Tests cover the affected behavior.
  • No dependency update is required.

@yuecidengyuecideng added agent Features related to agentic system enhancement New feature or request labels Aug 11, 2026
@yuecideng
yuecideng marked this pull request as ready for review August 11, 2026 16:46
CopilotAI lite review requested due to automatic review settings August 11, 2026 16:46
@greptile-apps

greptile-appsBot commented Aug 11, 2026

Copy link
Copy Markdown

Greptile Summary

The PR adds a strict MLLM-facing frontend that injects host-owned integration configuration, decodes constrained Expert Program JSON, enforces sequential and curated-call policies, and delegates compilation through the existing environment adapter.

  • Exposes decoding and compilation entry points through embodichain.agents.mllm.
  • Rejects model-controlled integrations, unsupported program structures, registered calls, and resource overrides.
  • Adds focused frontend tests and end-to-end equivalence coverage across MLLM, configuration, and Python skill paths.
  • Updates API documentation and applies postponed annotation evaluation to affected compatibility modules.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

FilenameOverview
embodichain/agents/mllm/expert_program.pyAdds strict decoding, policy enforcement, trusted integration injection, and compilation delegation without an eligible follow-up defect.
tests/agents/mllm/test_expert_program.pyCovers integration ownership, schema and node restrictions, resource overrides, strict JSON failures, and adapter compilation.
tests/gym/envs/expert_program/test_simulation_environment.pyExtends runtime-equivalence coverage to confirm that MLLM, decoded configuration, and Python skill frontends produce equivalent invocations and results.
embodichain/agents/mllm/init.pyExports the two new MLLM Expert Program entry points.
docs/source/api_reference/embodichain/embodichain.lab.gym.envs.expert_program.rstDocuments the constrained sequential MLLM frontend and its public functions.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
M[MLLM JSON response] --> P[Strict JSON parser]
H[Host integration] --> I[Inject trusted integration]
P --> I
I --> D[Canonical Expert Program decoder]
D --> V[MLLM policy validation]
V --> A[Environment adapter compile]
A --> C[Provider-free compiled program]
Loading

Reviews (5): Last reviewed commit: "Merge remote-tracking branch 'origin/mai..." | Re-trigger Greptile

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a strict MLLM-facing Expert Program frontend under embodichain.agents.mllm that (a) injects host-owned integration, (b) restricts the model to a curated schema-v1 surface, and (c) compiles via the existing Expert Program adapter/compiler path so model, config, and Python callers share one validation+lowering pipeline.

Changes:

  • Introduces decode_mllm_expert_program() / compile_mllm_expert_program() with a narrow policy (schema v1 only, curated call types only, no resource overrides, host-injected integration).
  • Adds focused unit tests for MLLM decode/compile behavior plus an integration-style equivalence test alongside existing simulation fixtures.
  • Updates API reference docs to expose the new MLLM frontend functions.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

Show a summary per file
FileDescription
tests/gym/envs/expert_program/test_simulation_environment.pyExtends simulation fixture tests to assert equivalence across Python calls, config decoding, and the MLLM frontend.
tests/agents/mllm/test_expert_program.pyNew unit tests covering MLLM integration injection, strict policy rejection cases, strict JSON parsing behavior, and compile-time scene reference errors.
embodichain/agents/mllm/expert_program.pyImplements strict MLLM decode + compile frontend that reuses existing Expert Program parsing/decoding/validation and adapter compilation.
embodichain/agents/mllm/init.pyExposes the MLLM frontend functions as the public embodichain.agents.mllm API surface.
embodichain/agents/init.pyIntroduces the embodichain.agents package initializer for agent-facing frontends.
docs/source/api_reference/embodichain/embodichain.lab.gym.envs.expert_program.rstDocuments and publishes the MLLM frontend functions in the API reference.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines +247 to +251
Raises:
TypeError: If ``adapter`` or ``integration`` has the wrong exact type.
ExpertProgramDecodeError: If JSON, schema, or MLLM policy validation
fails.
"""
Comment on lines +195 to +199
Raises:
TypeError: If ``integration`` is not an exact integration config.
ExpertProgramDecodeError: If JSON, schema, or MLLM policy validation
fails.
"""
Make endpoint bindings, timed command frames, transports, routing, safe holds, and profile lowering controller-agnostic. Preserve joint trajectories as optional feedback artifacts and add staged, same-address invocation revision semantics for mobile and whole-body safety.
…2c-runtime-endpoints
# Conflicts:
#	docs/design/declarative_expert_program_plan.md
#	embodichain/lab/sim/atomic_actions/primitives/coordinated_pickment.py
#	embodichain/lab/sim/atomic_actions/primitives/coordinated_placement.py
#	embodichain/lab/sim/atomic_actions/primitives/hand_over.py
#	embodichain/lab/sim/atomic_actions/primitives/move_end_effector.py
#	embodichain/lab/sim/atomic_actions/primitives/move_held_object.py
#	embodichain/lab/sim/atomic_actions/primitives/move_joints.py
#	embodichain/lab/sim/atomic_actions/primitives/pick_up.py
#	embodichain/lab/sim/atomic_actions/primitives/place.py
#	embodichain/lab/sim/atomic_actions/primitives/press.py
#	embodichain/lab/sim/skills/profiles.py
#	scripts/tutorials/atomic_action/assemble.py
#	scripts/tutorials/atomic_action/coordinated_pickment.py
#	scripts/tutorials/atomic_action/coordinated_placement.py
#	scripts/tutorials/atomic_action/hand_over.py
#	scripts/tutorials/atomic_action/move_end_effector.py
#	scripts/tutorials/atomic_action/move_held_object.py
#	scripts/tutorials/atomic_action/move_joints.py
#	scripts/tutorials/atomic_action/place.py
#	scripts/tutorials/atomic_action/press.py
@skywhite1024
skywhite1024force-pushed the feat/mllm-expert-program-frontend branch from 2df667f to 8bf6a50CompareAugust 21, 2026 09:35
@skywhite1024
skywhite1024force-pushed the feat/mllm-expert-program-frontend branch from 8bf6a50 to 2df667fCompareAugust 21, 2026 11:31
…rified-runtime
# Conflicts:
#	agent_context/topics/atomic-actions/atomic-actions.md
#	docs/source/overview/sim/atomic_actions/index.md
#	docs/source/overview/sim/semantic_skills.md
#	docs/source/tutorial/atomic_actions.rst
#	docs/source/tutorial/semantic_skills.rst
#	embodichain/lab/sim/atomic_actions/core.py
#	embodichain/lab/sim/atomic_actions/execution.py
#	embodichain/lab/sim/atomic_actions/plans.py
#	embodichain/lab/sim/atomic_actions/primitives/pick_up.py
#	embodichain/lab/sim/skills/runtime.py
#	tests/sim/atomic_actions/test_actions.py
#	tests/sim/atomic_actions/test_engine_per_env.py
#	tests/sim/skills/test_runtime.py
@yuecideng
yuecidengforce-pushed the feat/mllm-expert-program-frontend branch from 2df667f to 0b356f9CompareAugust 22, 2026 15:42
…-expert-program-frontend
# Conflicts:
#	docs/source/api_reference/embodichain/embodichain.lab.gym.envs.expert_program.rst
#	tests/gym/envs/expert_program/test_simulation_environment.py
Base automatically changed from feat/declarative-expert-program-runtime to mainAugust 24, 2026 08:25
…ram-frontend
# Conflicts:
#	docs/source/api_reference/embodichain/embodichain.lab.gym.envs.expert_program.rst
#	docs/source/api_reference/public_api.rst
#	embodichain/toolkits/graspkit/pg_grasp/__init__.py
#	tests/gym/envs/expert_program/test_simulation_environment.py
@yuecideng
yuecideng merged commit b7404d2 into mainAug 24, 2026
4 of 5 checks passed
@yuecideng
yuecideng deleted the feat/mllm-expert-program-frontend branch August 24, 2026 08:52
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agentFeatures related to agentic systemenhancementNew feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@yuecideng