External-depth-prior extension to Apple's ml-sharp Gaussian Splat predictor. Feeds a metric depth map (e.g. a Houdini cam_zdepth AOV, or any camera-space Z depth) into SHARP at inference time so the resulting splat has the correct metric scale and plausible parallax under modest camera moves.
Typical pipeline it was built for:
- 3D scene (Houdini / Blender / etc.) — beauty render + a camera-space depth AOV (Karma
cam_zdepth/hitPz) - Image model (an img2img pass, e.g. Nano Banana 2 / Gemini Flash Image) — a photoreal RGB from the beauty
- SHARP (this extension) — input = RGB + metric depth → metric Gaussian Splat
Use case: film / previz — the quality bar is that camera moves of a few degrees of orbit or a metre of dolly look parallactically plausible. The subclass + runtime hot-swap is the only modification path; Apple's ml-sharp is never edited.
The extension is its own package alongside an unmodified ml-sharp checkout (a sibling directory):
<parent>/
├── ml-sharp/ ← apple/ml-sharp @ cdb4ddc6, untouched
└── Sharp_Depth_Injection/ ← this repo
└── sharp_ext/
├── external_depth.py ← ExternalDepthGaussianComposer subclass
├── swap_composer.py ← runtime hot-swap into a built predictor
└── predict_with_depth.py
# 1. Clone Apple's ml-sharp at the pinned commit, as a sibling of this repo
git clone https://github.com/apple/ml-sharp.git ../ml-sharp
git -C ../ml-sharp checkout cdb4ddc6
# 2. Create a Python 3.13 venv inside this repo, and activate it
python -m venv .venv
# Windows: .venv\Scripts\Activate.ps1 macOS/Linux: source .venv/bin/activate# 3. Install ml-sharp's deps + ml-sharp itself + this extension
pip install -r ../ml-sharp/requirements.txt
pip install -e ../ml-sharp # so `import sharp` resolves
pip install -e .# so `import sharp_ext` resolvesCUDA torch: ml-sharp's requirements pull a CPU-only torch by default (silently runs on CPU). Force the CUDA build for your CUDA version, e.g. pip install --force-reinstall torch torchvision --index-url https://download.pytorch.org/whl/cu128, then verify python -c "import torch; print(torch.cuda.is_available())" prints True.
The SHARP checkpoint downloads automatically on first inference to ~/.cache/torch/hub/checkpoints/.
Set SHARP_TEST_IMAGE to any local RGB (falls back to ../ml-sharp/data/teaser.jpg if present). From the activated venv at the repo root:
python tests/test_decoder_stride.py # decoder grid resolution (stride 2 -> 768 grid)
python tests/test_self_consistency.py # feed SHARP's own depth back, expect near-identityDepth must be camera-space Z, in metres, positive forward (not ray distance, not world-Z). Sky / no-hit pixels that read 0 are remapped to a finite far distance by load_depth_exr, so SHARP doesn't plant fake near-camera Gaussians where the sky is.
importimageio.v3asiiofromsharp_extimportload_depth_exr, predict_image_with_depthfromtests._commonimportbuild_predictor, pick_devicedevice=pick_device()
predictor=build_predictor(device)
image=iio.imread("beauty.png") # H×W×3 uint8depth=load_depth_exr("depth.exr") # H×W float32, metres, +Z forward# f_px = focal_mm / horizontal_aperture_mm * image_widthgaussians=predict_image_with_depth(
predictor, image, f_px=877.71,
external_depth=depth, blend_alpha=0.4,
)See SINGLE_FRAME_INTEGRATION.md for a self-contained brief to embed the single-frame path in another app.
A FastAPI service wraps this library for image (+ metric depth EXR) → .ply over HTTP — see
service/README.md and service/API.md. It runs on :8765
(one model held in VRAM) and exposes three depth_methods: sharp, exr_pixel (per-pixel blend),
and exr_grade (remap SHARP's own predicted depth to the EXR's distribution). For splats prefer
exr_grade (grade_source=region, grade_curve=polynomial, grade_min_slope=1.0): it scales
SHARP's coherent geometry instead of injecting depth per pixel, which avoids the flying / smeared
gaussians exr_pixel produces at silhouettes.
Prerequisites: Python 3.13 (py -3.13 on PATH), git, and a recent NVIDIA driver. Then:
git clone https://github.com/gitcapoom/Sharp_Depth_Injection.git
cd Sharp_Depth_Injection
powershell -ExecutionPolicy Bypass -File setup.ps1 # install, then run# -AutoStart register a boot Scheduled Task instead of a foreground run# -NoRun install only# -TorchIndex https://download.pytorch.org/whl/cu121 # match an older CUDAsetup.ps1 wraps service/install.ps1: it clones ml-sharp at the pinned commit, builds the Python 3.13 venv, forces the CUDA torch build, installs OpenEXR, verifies torch.cuda.is_available(), then starts the service (or registers auto-start with -AutoStart). See service/README.md for management commands.
The released checkpoint applies a fixed internal NDC scaling, so an injected metric depth D lands at world-Z ≈ 3.4·D (both injection methods, uniform). A metric_rescale + far_cap fix was tried and reverted (far_cap flattened far geometry onto a wall). Correct the scale downstream in your consumer with a single uniform factor — divide positions and Gaussian scales by the measured factor (~3.4) — rather than in the shared service. Plain sharp (no injection) is unaffected. See CLAUDE.md for the full note.
Used in production: the extension + FastAPI service (service/serve.py) run as an auto-start service (:8765, one CUDA model held resident) driven over HTTP by downstream apps. Three depth methods ship (sharp / exr_pixel / exr_grade); the working splat recipe is exr_grade / region / polynomial / min_slope 1.0. Self-consistency + decoder-stride tests pass.