Reinforcement learning training and the environments it learns from, split into two processes and joined by a written protocol: WebSocket and msgpack, with a feedback return channel.
flowchart LR
subgraph S["plugrl-server · holds the policy"]
P["policy"] --- A["algorithm"]
end
subgraph C["plugrl-env-client · holds no policy"]
E["Gymnasium / MuJoCo / LIBERO"]
end
C -- "observation" --> S
S -- "action" --> C
C -- "feedback: reward, termination" --> S
The third arrow is the one that matters. Serving an inference model needs the first two; learning from what happened needs the third, and the protocol specifies it rather than leaving it to a convention.
Every experiment directory carries its data and a FINDINGS.md that states
what the result does not support.
| Question | Answer | |
|---|---|---|
| E1 | Do a training stack and an environment stack really conflict? | No. The claim this project was built on is disproved |
| E2 | Can anything but this codebase speak the protocol? | Yes - an 843-line C++ client with no third-party libraries |
| E7 | What does the boundary cost once packets leave the machine? | +0.52 ms on a 184 KiB observation |
| E10 | Is that cheap beside a VLA forward pass? | Yes - the split is 1.3-3.6% of a step |
| E11 | Can a real VLA be trained through it, and does it help? | Trained, not helped - below |
| E12 | Does a rollout machine really need CUDA, as E1 concluded? | No - that was a packaging default. LIBERO's env client goes 7.8G to 3.4G, no nvidia wheels |
| E13 | And without a GPU to render on? | Yes, at 1.91x - ten clients rendering on the CPU, 30 of 30 episodes successful |
The training server is 6.5G and wants a GPU. The environment side does not have to be either: 3.4G, no CUDA, no GPU driver, rendering on the CPU at 1.91x the wall clock. That is a different class of machine - it fits where the trainer does not.
E1 had measured this and concluded the opposite, because a default pip install torch brings CUDA along whether or not anything uses it. E12 reproduced E1's
row exactly before changing one pin, so the two sets of numbers are comparable;
E1's sentence was narrowed in place with a dated note rather than deleted.
A full-size pi0.5 ran end to end through the boundary on LIBERO - inference,
feedback and FPO training - and the server's record of episodes and steps
reconciles exactly with the clients'. As a control, the unmodified checkpoint
scored 99 of 100 on libero_spatial and 185 of 200 on libero_10, against
openpi's published 98.8 and 92.4.
Then one FPO iteration took the hardest task from 26 of 50 to 0 of 50, and the run is incomplete at one iteration of ten: a second learn step does not fit beside the optimizer state the first one allocates on a 24 GB card. The predictions were registered before the run, and one of them is falsified.
That is what the data says, so that is what is written down. One experiment disproved the assumption the project was founded on, and two withdrew earlier claims of our own.
| plugrl-server | Training side: policy, algorithm, checkpoints, and the experiments |
| plugrl-env-client | Environment side: steps envs, asks for actions, returns feedback |
| plugrl-protocol | The specification, its checkable clauses as tests, and two reference env clients |
| plugrl.github.io | Documentation, in English and 中文 |
Start at the documentation - the quickstart trains FPO on HalfCheetah with no GPU and nothing to download.
PlugRL is built by Chenhao Lu, Zuo Gou and Zilin Kang.