SpotObserver is a C++/CUDA library for streaming and processing camera data from
Boston Dynamics Spot robots. It is built
as a native Unity plugin (SpotObserverLib.dll) and exposes a C API for:
- Connecting to one or more Spot robots and streaming RGB + depth feeds from any of the onboard cameras (back, front-left, front-right, left, right, hand)
- A GPU vision pipeline (for depth completion at the moment) powered by LibTorch and ONNX Runtime
- Low-overhead handoff of images into Unity via CUDA/DX12 interop.
- Per-frame body-to-world transforms alongside the images
The full C API is declared in include/spot-observer.h.
A Pythonic implementation of the same functionality lives in
PySpotObserver/ — see its README and
QUICKSTART. It provides a clean, type-hinted API with
sync and async usage, context-managed connections, YAML configuration, and an optional
ONNX vision pipeline:
frompyspotobserverimportSpotConfig, SpotConnection, CameraTypewithSpotConnection(SpotConfig(robot_ip="192.168.80.3", username="...", password="...")) asconn:
stream=conn.create_cam_stream()
stream.start_streaming(CameraType.FRONTLEFT|CameraType.FRONTRIGHT)
rgb_images, depth_images, body_T_worlds=stream.get_current_images()| Path | Contents |
|---|---|
include/ | Public C API header (spot-observer.h) |
src/ | Library implementation (connection, vision pipeline, CUDA kernels, Unity/DX12 interop) |
PySpotObserver/ | Python bindings and examples |
extern/ | Third-party dependencies (Spot C++ SDK, OpenCV, LibTorch, ONNX Runtime) |
tests/ | Integration tests (integ-test, integ-test-dx12) and standalone tests |
scripts/ | Utility scripts (e.g. timing statistics) |
- Windows 10/11 x64 with Visual Studio 2022 (MSVC v143 toolset)
- CMake 3.28+
- NVIDIA CUDA Toolkit (currently only tested on Ampere and Ada GPUs, compute 8.6 / 8.9)
- A Unity Editor installation (the CUDA/DX12 interop module uses Unity's Native Plugin API headers;
the build auto-detects the newest Unity Hub install — override with
-DUnityPluginAPI_INCLUDE=<Unity>/Editor/Data/PluginAPIif needed)
All third-party dependencies live under extern/. Follow
extern/README.md for the detailed steps.
Use CMAKE_PREFIX_PATH (pointing at the vcpkg installed tree), notCMAKE_TOOLCHAIN_FILE:
cmake -B build \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_PREFIX_PATH="<vcpkg-abs-path>/installed/x64-windows" \
-DCMAKE_FIND_PACKAGE_PREFER_CONFIG=TRUE
cmake --build build --config ReleaseThis produces SpotObserverLib.dll and automatically copies all required runtime DLLs
(vcpkg, OpenCV, LibTorch, ONNX Runtime) next to the built binaries. Set
-DCOPY_DLLS=OFF to disable the automatic copying.
Useful options:
| Option | Default | Effect |
|---|---|---|
BUILD_TESTS | ON | Build the test executables under tests/ |
INSTALL_TESTS | ON | Install test executables |
COPY_DLLS | ON | Copy required DLLs next to the built library |
cmake --install build --config ReleaseBy default this installs the library, public headers, and runtime DLLs into
<repo>/install. Override with -DCMAKE_INSTALL_PREFIX=<path> at configure time.
With BUILD_TESTS=ON (the default), the build also produces:
tests/integ-test— end-to-end streaming test against a real robottests/integ-test-dx12— DirectX 12 interop / readback testtests/standalone-tests— smaller isolated tests
- Robot credentials are passed to
SOb_ConnectToSpot()at runtime; nothing is hardcoded in the library. - The Boston Dynamics C++ SDK is still in beta