Scene-Level Heterogeneous Physics Simulation with 3D Gaussian Splats

Official code release for Scene-Level Heterogeneous Physics Simulation with 3D Gaussian Splats.

Project Page | arXiv

Overview

This project couples Genesis physics simulation with NVIDIA Omniverse RTX rendering for heterogeneous scenes that combine 3D Gaussian Splats, deformable objects, fluids, cloth, and rigid geometry.

The paper discusses a UE5 rendering backend for high-end production rendering. This release provides an Omniverse RTX backend for easier setup and reproducibility; the heterogeneous simulation components can also be connected to UE5 for more advanced water and production rendering effects.

This repository provides two runnable examples:

Script	Demo	Output
scripts/run_donut_chocolate.sh	3DGS donuts with viscous chocolate drizzle	outputs/donut_chocolate/omniverse_render.mp4
scripts/run_cloth_statue.sh	Textured cloth sliding over a 3DGS statue scene	outputs/cloth_statue_3dgs/omniverse_render.mp4

Preview media are included under media/:

• media/donut_chocolate_preview.mp4
• media/donut_chocolate_preview.gif
• media/cloth_statue_3dgs_preview.mp4
• media/cloth_statue_3dgs_preview.gif

Repository Layout

third_party/genesis-world/ Bundled Genesis simulator source used by the demos
src/genesis_omniverse_bridge/ Genesis-to-Omniverse bridge implementation
src/genesis_omniverse_bridge/omni/
Omniverse Kit launch and RTX rendering code
scripts/ Runnable demo entry points
configs/ Demo simulation and rendering configurations
media/ Lightweight preview videos and GIFs
assets/ Demo assets after extracting the asset archive

Quick Start

Clone the repository:

git clone https://github.com/xiaoyangliu137/raf.git scene-level-heterogeneous-physics-simulation
cd scene-level-heterogeneous-physics-simulation

Create and activate the Python environment:

conda create -n scene-physics-3dgs python=3.11 -y
conda activate scene-physics-3dgs

Install PyTorch and the Python packages:

pip install torch --index-url https://download.pytorch.org/whl/cu124
pip install -e third_party/genesis-world
pip install -e .

The third_party/genesis-world package is the bundled Genesis simulator source used by these demos. Do not replace it with the official upstream Genesis package when reproducing these examples.

Download the demo asset archive from Google Drive, then extract it into the repository root:

unzip ~/Downloads/scene-level-heterogeneous-physics-simulation-assets.zip -d .
ls assets/donuts assets/cloth_statue

Set up NVIDIA Omniverse Kit. This project uses Kit as the RTX renderer runtime; the rendering code is included in this repository, but the NVIDIA Kit runtime must be installed separately.

Requirements:

• Linux x86_64.
• NVIDIA RTX GPU with a recent NVIDIA driver.
• An Omniverse Kit runtime containing kit and apps/omni.app.viewport.kit.

Recommended setup using NVIDIA's official Kit App Template:

cd ..
git clone https://github.com/NVIDIA-Omniverse/kit-app-template.git
cd kit-app-template
./repo.sh template new
./repo.sh build
export OMNI_KIT_ROOT="$(pwd)/_build/linux-x86_64/release/kit"test -x "$OMNI_KIT_ROOT/kit"test -f "$OMNI_KIT_ROOT/apps/omni.app.viewport.kit"export OMNI_KIT_ACCEPT_EULA=YES
cd ../scene-level-heterogeneous-physics-simulation

When ./repo.sh template new prompts for a template, choose an Application template and select Kit Base Editor. The ./repo.sh build step creates the local Kit runtime used by this project.

Alternative setup using NGC:

• Open the Omniverse Kit SDK Linux package.
• Install the NGC CLI and run ngc config set.
• Use the download command shown on the NGC Kit SDK page for the current Linux Kit SDK version.
• Extract/build the SDK package, then set OMNI_KIT_ROOT to the folder that contains both kit and apps/omni.app.viewport.kit.

If Kit is already installed somewhere on the machine, locate it and set OMNI_KIT_ROOT manually:

KIT_EXE=$(find "$HOME" /opt /tmp -type f -name kit -path '*kit*'2>/dev/null | head -n 1)export OMNI_KIT_ROOT="$(dirname "$KIT_EXE")"test -x "$OMNI_KIT_ROOT/kit"test -f "$OMNI_KIT_ROOT/apps/omni.app.viewport.kit"export OMNI_KIT_ACCEPT_EULA=YES

Useful official references:

• Omniverse Kit SDK Overview
• Omniverse Kit Manual
• NVIDIA Omniverse Libraries

The NVIDIA runtime itself is not vendored in this repository; the bundled Genesis source and the Omniverse rendering bridge code are included.

Omniverse Version and 3DGS Compatibility

Use the newest Omniverse Kit SDK available for your machine when possible. Native 3D Gaussian Splat rendering in this project relies on OpenUSD ParticleField support and Omniverse RTX Particle Field imaging. OpenUSD 26.03 introduced the ParticleField schema family for 3D Gaussian Splats, and Omniverse RTX 110.1 documents native Particle Field rendering with path tracing, shadows, reflections, and refractions. In practice, use a Kit/RTX build that includes the OpenUSD ParticleField3DGaussianSplat schema and the omni.usd.schema.usd_particle_field extension.

The launcher in this repository enables the required extensions explicitly:

omni.usd
omni.usd.schema.usd_particle_field
omni.hydra.rtx
omni.kit.renderer.capture

The renderer also handles schema namespace differences across Kit builds. It first tries UsdParticleField.ParticleField3DGaussianSplat, then UsdVol.ParticleField3DGaussianSplat, and finally authors a ParticleField3DGaussianSplat prim directly. This compatibility path helps with Kit/OpenUSD version differences, but it does not replace a renderer that lacks ParticleField imaging support.

If the latest Kit starts correctly on your GPU, use it. If the latest Kit fails because of driver/runtime compatibility on your system, use an older Kit runtime that still includes ParticleField support and run a smoke test:

FRAME_LIST=0,last ./scripts/run_donut_chocolate.sh

For driver-limited systems where Path Tracing is unstable but RTX rendering works, try the real-time RTX mode without changing the YAML files:

RENDER_OVERRIDE='{"path_tracing": false, "spp": 128, "frame_settle_frames": 64}' \
FRAME_LIST=0,last \
./scripts/run_donut_chocolate.sh

This fallback uses RaytracedLighting instead of PathTracing. It is useful for compatibility testing, while the default donut/chocolate demo remains configured for higher-quality path-traced rendering.

Relevant ParticleField references:

• OpenUSD ParticleField3DGaussianSplat schema
• Omniverse Gaussian Splats / Particle Fields
• Omniverse RTX 110.1 release highlights

Select a GPU and run the demos:

export CUDA_VISIBLE_DEVICES=0
./scripts/run_donut_chocolate.sh
./scripts/run_cloth_statue.sh

The output videos are written to:

outputs/donut_chocolate/omniverse_render.mp4
outputs/cloth_statue_3dgs/omniverse_render.mp4

For a short smoke test, render only three frames:

FRAME_LIST=0,60,last ./scripts/run_donut_chocolate.sh
FRAME_LIST=0,120,last ./scripts/run_cloth_statue.sh

Remove FRAME_LIST for full video rendering.

Omniverse Rendering Code

The Omniverse rendering implementation is included in this repository:

src/genesis_omniverse_bridge/omni/run_kit.py
src/genesis_omniverse_bridge/omni/render_demo.py
src/genesis_omniverse_bridge/gaussian.py
src/genesis_omniverse_bridge/donut_3dgs.py

run_kit.py launches Omniverse Kit in headless RTX mode. render_demo.py builds the USD stage, creates the camera, lights, mesh tracks, chocolate surface mesh, cloth mesh, and OpenUSD ParticleField 3DGS prims, then renders PNG frames. The final MP4 is encoded by the Python CLI after rendering.

Each demo run performs:

Genesis simulation
-> genesis_native.mp4
-> simulation_tracks.npz
-> manifest.json
Omniverse RTX rendering
-> stage.usdc
-> rtx_frames/
-> omniverse_render.mp4

Run Demos

The main entry points are:

./scripts/run_donut_chocolate.sh
./scripts/run_cloth_statue.sh

Each run writes:

outputs/<demo>/
├── genesis_native.mp4
├── manifest.json
├── omniverse_render.mp4
├── rtx_frames/
├── simulation_tracks.npz
└── stage.usdc

Preview

The animations below show representative outputs from the default configurations. Click a preview to open the higher-quality MP4.

3DGS donuts with viscous chocolate drizzle

Textured cloth sliding over a 3DGS statue scene

Configuration

Default demo parameters are in:

configs/donut_chocolate.yaml
configs/cloth_statue_3dgs.yaml

Commonly edited sections include:

• simulation: timestep, duration, gravity, and solver settings.
• camera: resolution, pose, field of view, and orbit settings.
• donuts: initial object placement and material settings.
• donut_3dgs_render: 3DGS asset paths and render alignment.
• emitter: chocolate source position, motion, and material settings.
• cloth: cloth size, material, texture, and initial pose.
• gaussian: 3DGS scene path and display parameters.
• render_settings: Omniverse resolution, sampling, bounces, and lights.

Paths in YAML files are relative to the config file location.

Citation

If you find this code useful, please cite:

@inproceedings{liu2026scenelevel,
title = {Scene-Level Heterogeneous Physics Simulation with 3D Gaussian Splats},
author = {Liu, Xiaoyang and Wu, Shangzhe and Han, Kai},
booktitle = {CVPR Findings},
year = {2026}
}

License

The source code is released under the MIT License. See LICENSE for details. Demo assets and external runtimes are governed by their own licenses.