Implement online streaming inference pipeline (Algorithm 2) with real-time web dashboard

[jflinte]

Motivation

The existing inference scripts (splat_inference.py and demo/run_demo.py) process videos offline by packaging the entire sequence into chunked 2-frame pairs without maintaining a continuous canonical 3DGS state or performing dynamic Gaussian pruning.

Algorithm 2 ("Online Inference") in the StreamSplat paper defines a true online streaming architecture that maintains bounded $O(1)$ VRAM memory over streams of arbitrary length by:

1. Maintaining a persistent canonical Gaussian set $\tilde{\mathcal{G}}(t)$ and cached feature embedding $h_{k-1}$.
2. Propagating active forward Gaussians, fusing new backward Gaussians, and pruning dead Gaussians ($\alpha_g(t_k) \le 0$).

Additionally, users require an effective, interactive way to monitor the reconstruction in real time with visual feedback and telemetry.

Proposed Approach

• Algorithm 2 Streaming Engine (stream_online.py / model/stream_engine.py):
  • Implement stateful streaming: cache previous embedding $h_{k-1}$ and DINOv2 condition features $f_{k-1}$ in VRAM (~9 MB overhead).
  • For each incoming frame $I_k$: estimate depth $D_k$, encode $h_k$, run dynamic decoder on $(h_{k-1}, h_k)$ to generate forward field $d_{k-1}^+$ and backward field $d_k^-$.
  • Update motion trajectories of persistent Gaussians, fuse incoming backward Gaussians, and prune Gaussians where $\alpha(t_k) < \epsilon$.
  • Rasterize intermediate timestamps and novel viewpoints on the fly.
• Interactive Real-Time Web Dashboard:
  • Lightweight local server (FastAPI + WebSockets) serving a modern frontend.
  • Live 3-panel monitor: Incoming RGB stream ($I_k$), Depth Anything V2 depth estimation ($D_k$), and Dynamic 3DGS reconstruction ($I_t$).
  • Interactive 3D viewport: Orbit/pan/zoom camera to inspect the 3D scene from novel views during streaming.
  • Live HUD telemetry: Active Gaussian count, pruning rate, latency/FPS breakdown (encoder, decoder, rasterizer), and VRAM usage.

Acceptance Criteria

• Online streaming pipeline faithfully executes Algorithm 2 with stateful persistent Gaussians and online pruning.
• VRAM footprint remains constant ($O(1)$) over arbitrarily long video streams.
• Interactive web dashboard streams live RGB, depth, and 3DGS renders with novel-view orbit camera control.
• Live telemetry displays active Gaussian count, pruning metrics, and latency breakdown.