A complete, production-ready platform for building, training, and deploying large-scale multi-modal world models for autonomous robotics and embodied AI
Empowering the next generation of intelligent robots with advanced predictive control and real-time decision making
OpenControl is specifically engineered for real-world robotics applications, providing:
- Real-Time Robot Control: Sub-30ms inference for live robotic systems
- Multi-Modal Perception: Vision, tactile, proprioceptive, and audio sensor fusion
- Predictive Planning: Advanced world models for robot motion planning
- Safety-Critical Systems: Built-in safety constraints and failure detection
- Hardware Integration: Direct interfaces to popular robotics platforms
- Sim-to-Real Transfer: Seamless deployment from simulation to physical robots
- Visual Model Predictive Control (MPC): High-performance real-time control with Cross-Entropy Method optimization
- Transformer-Based World Models: State-of-the-art architecture with rotary positional embeddings (RoPE)
- Multi-Modal Sensor Fusion: RGB, depth, tactile, IMU, and proprioceptive data integration
- Real-Time Inference: Optimized for robotics control loops (30-1000Hz)
- Safety Systems: Built-in collision avoidance and constraint satisfaction
- Continuous Learning: Online adaptation and few-shot learning capabilities
- Containerized Deployment: Docker containers for edge and cloud deployment
- Distributed Training: Multi-GPU and multi-node training with mixed precision
- Interactive Dashboard: Rich terminal interface with live robot monitoring
- Comprehensive Metrics: Advanced evaluation suite for robot performance assessment
- Modern Tooling: Built with uv, tox, and modern Python development practices
- Cloud Integration: Support for AWS, GCP, and Azure robotics services
OpenControl's architecture is designed specifically for robotics applications:
┌─────────────────────────────────────────────────────────────────────────────┐
│ ROBOTICS CONTROL LOOP │
├─────────────────────────────────────────────────────────────────────────────┤
│ Sensors → Perception → World Model → Planning → Control → Actuators │
└─────────────────────────────────────────────────────────────────────────────┘
┌─────────────────────┐ ┌─────────────────────┐ ┌─────────────────────┐
│ Sensor Fusion │───▶│ World Model │───▶│ Safety Monitor │
│ (Vision/Tactile/IMU)│ │ (Transformer) │ │ (Constraints) │
└─────────────────────┘ └─────────────────────┘ └─────────────────────┘
│ │
▼ ▼
┌─────────────────────┐ ┌─────────────────────┐ ┌─────────────────────┐
│ Robot Hardware │◀───│ Visual MPC │◀───│ Motion Planner │
│ (Arms/Grippers/Base)│ │ (Real-time Control) │ │ (Trajectory Gen) │
└─────────────────────┘ └─────────────────────┘ └─────────────────────┘
# Clone the repository
git clone https://github.com/llamasearchai/OpenControl.git
cd OpenControl
# Create virtual environment with uv
uv venv
source .venv/bin/activate # On Windows: .venv\Scripts\activate# Install dependencies
uv pip install -e ".[dev,robotics]"# Run tests to verify installation
pytest tests/# For UR5/UR10 robots
opencontrol robot connect --type universal_robots --ip 192.168.1.100
# For Franka Panda
opencontrol robot connect --type franka --ip 172.16.0.2
# For custom robots via ROS
opencontrol robot connect --type ros --namespace /robot_arm# Train on robot demonstration data
opencontrol train --config configs/robots/manipulation.yaml \
--data-path data/robot_demos/ \
--robot-type franka_panda
# Train with simulation data
opencontrol train --config configs/robots/sim_to_real.yaml \
--sim-data data/simulation/ \
--real-data data/robot_demos/# Start robot control server
opencontrol serve --robot-config configs/robots/franka_config.yaml \
--model checkpoints/best_model.pt \
--control-freq 100 # 100Hz control loop# Launch interactive robot dashboard
opencontrol dashboard --robot-ip 192.168.1.100fromopencontrolimportRobotController, WorldModel# Initialize robot controllerrobot=RobotController(
robot_type="franka_panda",
model_path="checkpoints/manipulation_model.pt",
control_frequency=100# Hz
)
# Load pre-trained world modelworld_model=WorldModel.from_pretrained("opencontrol/manipulation-v1")
# Execute pick and place tasksuccess=robot.execute_task(
task="pick_and_place",
target_object="red_cube",
destination="blue_box",
world_model=world_model
)- Universal Robots: UR3, UR5, UR10, UR16
- Franka Emika: Panda, Research 3
- Kinova: Gen2, Gen3, MOVO
- ABB: IRB series, YuMi
- KUKA: iiwa, KR series
- Custom Arms: Via ROS/ROS2 interface
- TurtleBot: 2, 3, 4
- Clearpath: Husky, Jackal, Ridgeback
- Boston Dynamics: Spot (via SDK)
- Custom Platforms: Via ROS/ROS2 navigation stack
- Robotiq: 2F-85, 2F-140, 3F series
- Schunk: EGP, EGK series
- OnRobot: RG2, RG6, VG10
- Custom Grippers: Via GPIO/Modbus/EtherCAT
- Cameras: Intel RealSense, Azure Kinect, Zed
- Force/Torque: ATI, Robotiq FT 300
- Tactile: Digit, GelSight, TacTip
- LiDAR: Velodyne, Ouster, Sick
OpenControl is optimized for real-world robotics performance:
- Data Throughput: 1000+ robot episodes/hour on 8x A100 GPUs
- Model Scaling: Supports models up to 70B+ parameters
- Memory Efficiency: Gradient checkpointing and mixed precision training
- Distributed Training: Linear scaling across multiple nodes
- Control Latency: <30ms end-to-end (perception to action)
- Throughput: 1000+ inferences/second on RTX 4090
- Memory Usage: <8GB VRAM for 7B parameter models
- CPU Inference: Real-time performance on modern CPUs
- Pick Success Rate: >95% on standard benchmarks
- Manipulation Precision: <2mm positioning accuracy
- Collision Avoidance: 99.9% success rate in cluttered environments
- Adaptation Speed: <10 demonstrations for new tasks
- Assembly Line Integration: Automated part assembly and quality control
- Flexible Manufacturing: Rapid reconfiguration for different products
- Quality Inspection: Vision-based defect detection and sorting
- Human-Robot Collaboration: Safe shared workspace operations
- Household Assistance: Cleaning, cooking, and maintenance tasks
- Healthcare Support: Patient assistance and medical device operation
- Hospitality: Food service and customer interaction
- Elder Care: Mobility assistance and health monitoring
- Manipulation Research: Advanced grasping and dexterous manipulation
- Navigation Studies: Indoor and outdoor autonomous navigation
- Human-Robot Interaction: Natural language and gesture-based control
- Multi-Robot Systems: Coordination and swarm robotics
- Precision Agriculture: Crop monitoring and selective harvesting
- Environmental Monitoring: Data collection in remote locations
- Search and Rescue: Autonomous exploration and victim detection
- Construction: Automated building and infrastructure maintenance
- Installation Guide - Complete setup instructions
- Robotics Tutorial - Step-by-step robot integration
- Quick Start Examples - Ready-to-run code examples
- Hardware Setup - Robot and sensor configuration
- World Models - Understanding predictive models
- Visual MPC - Model predictive control systems
- Multi-Modal Learning - Sensor fusion techniques
- Safety Systems - Constraint satisfaction and collision avoidance
- Training Guide - Model training and optimization
- Data Collection - Robot demonstration recording
- Configuration - System configuration options
- Debugging - Troubleshooting and diagnostics
- Deployment Guide - Production deployment strategies
- Docker Setup - Containerized deployment
- Cloud Integration - AWS, GCP, and Azure setup
- Monitoring - System monitoring and logging
- Robot API - Robot control interfaces
- Model API - World model interfaces
- Metrics API - Evaluation and monitoring
- Utils API - Utility functions and helpers
# Clone and setup
git clone https://github.com/llamasearchai/OpenControl.git
cd OpenControl
# Setup development environment
uv venv
source .venv/bin/activate
uv pip install -e ".[dev,robotics]"# Install pre-commit hooks
pre-commit install
# Run tests
pytest tests/# Install development dependencies
uv pip install -e ".[dev]"# Run tests with tox
tox
# Run specific test environments
tox -e py311 # Python 3.11 tests
tox -e lint # Linting
tox -e type-check # Type checking
tox -e benchmark # Performance benchmarks
tox -e robotics # Robot integration tests# Format code
tox -e format
# Run all checks
tox -e lint
tox -e type-check
# Run robot-specific tests
pytest tests/robotics/ -vWe especially welcome contributions in:
- New robot platform integrations
- Advanced control algorithms
- Safety system improvements
- Simulation environments
- Hardware driver optimizations
See our Contributing Guide for detailed guidelines.
If you use OpenControl in your robotics research, please cite:
@software{opencontrol2024,
title={OpenControl: Advanced Multi-Modal World Model Platform for Robotics},
author={Jois, Nik},
year={2024},
url={https://github.com/llamasearchai/OpenControl},
version={1.0.0}
}This project is licensed under the Apache License 2.0 - see the LICENSE file for details.
- Built with modern Python tooling: uv, tox, and pytest
- Leverages PyTorch ecosystem for deep learning
- Inspired by advances in robotics, large language models, and embodied AI
- Special thanks to the robotics community for feedback and contributions
Nik Jois - nikjois@llamasearch.ai
- Project Link: https://github.com/llamasearchai/OpenControl
- Documentation: https://opencontrol.readthedocs.io/
- Issues: https://github.com/llamasearchai/OpenControl/issues
- Discussions: https://github.com/llamasearchai/OpenControl/discussions
- Discord: Join our robotics community server
- Mailing List: Subscribe for updates and announcements
- Workshops: Regular training sessions and tutorials
- Consulting: Enterprise support and custom development
Built for the Future of Robotics
Empowering researchers, engineers, and companies to build the next generation of intelligent robots
Made by the OpenControl team