Uh oh!
There was an error while loading. Please reload this page.
Beta: camera streaming, shared memory, and the beta protocol - #5
Open
jonathanembleyriches wants to merge 20 commits into
Open
Beta: camera streaming, shared memory, and the beta protocol#5jonathanembleyriches wants to merge 20 commits into
jonathanembleyriches wants to merge 20 commits into
Conversation
…XML model An unloadable MJB (cross-version) now falls back to building the local model from the handshake's mjcf_compiled + vfs_assets instead of leaving art.joints/art.actuators silently empty. Removes the local_model=False bypasses from the dashboard and integration conftest.
…ulation Thin wrapper over the new UE set_camera_delay op: per-camera delay_s / jitter_s / clock / seed plus capture-rate knobs (on_state_change, max_fps). Frames arrive pre-delayed over the existing stream, so no client-side buffering.
The SHM RPC region (req/rep.shm + kernel events) lives on a static session, but the client re-derived it from the per-PIE camera/state stream dir. Requests landed in a directory UE's RPC worker never serviced, so every SHM step stalled its full timeout (~5s) then fell back to ZMQ. Read the advertised shm_rpc block (paths + event names) and wire the RPC region separately from the stream dir. SHM step RTT drops from ~5000ms to ~1.5ms.
- client: transport=auto, SHM handshake contract, camera delay tracking, camera_frame_cond - transports: ZMQ state-loop reconnect, SHM wrong_transport/retry-safe ops, resolve_endpoint - articulation: zmq_endpoint/zmq_topic properties - farm: URLabPool discovery/leasing, launcher, model upload - policy: state_stream, auto_grasp, lerobot_runner stubs - cleanup: remove urlab_tools (ROS now in-plugin), drop ros2 extra, update deps
…r to an OS-assigned port The step_ok reply renamed per_articulation -> arts and entities -> scene; six tests still hand-built replies with the old keys. Requests still use per_articulation, so those assertions are unchanged. MockStepServer used a fixed 55900+ counter band, which sits inside the Windows dynamic port range and intermittently collided with an outbound socket in the suite.
attach_puppet_simulation binds an external MjModel before the URLab
scene exists — the level is loaded, the MJCF imported and the actor
spawned afterwards — so the two models can diverge before the first
push, and a puppet step is just a raw qpos/qvel/ctrl vector with no way
to notice.
Every handshake carrying an MJB is now compared against the attached
model. Dimension mismatches always raise, since a push transmits exactly
nq/nv/nu floats and no reading of differing counts is benign. Name and
joint-ordering differences follow puppet_drift_check ('error' | 'warn' |
'off'), defaulting to warn because a scene may legitimately carry bodies
the source model never had. sim.start absorbs a fresh handshake once PIE
is up, so the check covers the post-import state, not just connect.… API Puppet mode pauses the server's camera publishers (the step server owns cadence there), so `camera_query="latest"/"fresh"` wait on streams that never tick and no typed call could produce a frame. The server's inline path renders from the state the step just applied and waits on that capture's readback serial, which is the stronger correspondence guarantee -- but `include_cameras` was never forwarded onto the wire by either step path, so it was unreachable except by hand-building the RPC. Add `camera_query="sync"` to request it. The reply decode already existed; only the request side was missing. Two fixes in that decode, both of which the sync path now exercises: - It searched per-articulation cameras only, so a scene-level camera -- a worldbody <camera> in the imported MJCF, which is what an external simulation exports -- had its pixels silently discarded. Use the same lookup as `get_camera`, which also covers `global_cameras`. - It sized the buffer from the view's registered resolution and dropped the frame on a mismatch. Prefer the payload's own width/height, which are what the bytes were actually sized by.
The plugin builds MuJoCo from source at 3.11.1 and 3.11.0 is the newest PyPI release, so the client pins one patch behind on purpose: the handshake compares major.minor, and a patch-level skew falls back to the compiled-XML model instead of the version-locked MJB. mujoco-warp moves with it, or mjlab's import-time enum lookup fails and the runner is silently absent. Asserted in tests so the two pins cannot drift apart unnoticed.
The six-way sweep gated SHM on Linux and said the Windows kernel-event path was covered by separate platform runs. There are none, so half the matrix never ran on the platform most of the development happens on. The client implements POSIX futex and Windows events alike; gate on that.
The server refuses actuator writes from a client holding no claim, which `claim_control` was added for and the policy runners already do. The integration fixtures never caught up, so twelve live tests failed on `not_control_owner` rather than on anything they were testing.
An articulation's `prefix` is its ActorId, which the server reports so topics and control-ownership keys stay stable across spawns. The compiled model names its elements from the UE object name instead, and the server sends that as `actor_name`. Matching model names against `prefix` found nothing, so an articulation came back with no actuators, joints or sensors -- an inert robot with no error anywhere.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
synccapture reachable from the typed step API, and server-side latency emulation viaruntime.set_camera_delay.Motivation
Camera frames used to ride in the step reply, which bloated it and stalled high-rate puppet stepping. Serving them from async streams decouples camera rate from step rate. The rest is the client catching up to the plugin's beta protocol.
Notable fixes
ctrland the server refusesctrlfrom a client holding no claim, so the Step button answerednot_control_ownerand nothing moved.connect()ran on the UI thread, and with a camera in the scene its last act starts a broadcast and opens a SUB socket per camera — seconds of work with the window frozen.set_camera_streaming,list_actors,find_actors,get_actor_bounds,actor_hierarchy,snapshot. These walk the whole world on the game thread and outgrew the 5 s default.<general>actuators are accepted on the wire. It is MJCF's own actuator element and the one every other kind is shorthand for; mjlab models author it directly, and 29 of them came back typeless.Test evidence
The one skip is honest: the golden arm uses
<position>actuators, so it has no UE-side controller for the PD-gain test to exercise.Compatibility
Requires the plugin beta (
URLab-Sim/UnrealRoboticsLab), which this branch's protocol tracks.Supersedes
Closes the earlier external-puppet and MuJoCo-alignment PRs; their commits are carried here with authorship intact.