Visual Positioning System for Drones

GPS-denied geolocalization for drones, in the spirit of Theseus. A downward-facing camera frame plus basic flight telemetry (altitude + IMU attitude) is cross-referenced against a pre-loaded georeferenced basemap to recover the drone's geographic coordinates — with no GPS and no internet at runtime.

The map you localize against is a local GeoTIFF file on disk. There is no maps API, no cloud service, and no network connection required at runtime. The only time the network is used is pip install.

How it works

For every camera frame the engine runs a four-step pipeline:

1. Orthorectify — warp the oblique camera image into a north-up, metric, top-down patch using attitude + altitude + camera intrinsics (a ground-plane homography).
2. Predict & crop — a constant-velocity motion filter predicts where the drone is; an equally-scaled patch is cropped from the basemap there.
3. Match — classical OpenCV feature matching (ORB/SIFT + RANSAC) aligns the two patches into a 2D similarity transform.
4. Geo-solve & fuse — the transform yields a lat/lon fix plus heading and a confidence score; implausible jumps are gated out and the trusted fixes are fused.

All matching math happens in the basemap's projected CRS (UTM-like meters), so distances are true ground meters. WGS84 lat/lon appears only at the input and output edges.

The whole system is exposed through a single object: Estimator.estimate(frame) -> PositionEstimate. The CLI drives it; a future ROS2 node will drive the exact same engine without any changes to the core.

Requirements

• Python 3.12 — the geospatial/CV stack (rasterio, opencv) has no wheels for Python 3.13/3.14, which may be your system default.
• Windows: Python 3.12 via python.org or winget install Python.Python.3.12. Open a new terminal after installing so the py launcher is on your PATH.
• macOS/Linux: brew install python@3.12 or your distro's package manager.

Install

Clone the repo and create a virtual environment with Python 3.12:

# Windows (PowerShell)
py -3.12-m venv .venv
.\.venv\Scripts\python.exe-m pip install -e ".[dev]"

# macOS / Linux
python3.12 -m venv .venv
.venv/bin/pip install -e ".[dev]"

The [dev] extra installs pytest. All other dependencies are in the core package. No extras are needed to run the CLI and tests.

Quickstart

One command does everything — synthesize a basemap, fly a synthetic flight, localize it, and score the result:

# Windows
.\.venv\Scripts\vps.exe demo --out demo_out
# macOS / Linux
.venv/bin/vps demo --out demo_out

Expected output (numbers vary slightly by machine):

 frames : 665
localized : 665 (100.0%)
trusted : 665 (100.0%)
position error: median 1.011 m | mean 1.1 m | p95 2.115 m | max 3.172 m
heading error : median 0.026° | p95 0.089°

The trusted line shows how many fixes passed all quality gates and are safe to inject as GPS replacements (see The success flag below).

All CLI commands

# Synthesize a georeferenced basemap (saved as a GeoTIFF)
vps make-basemap --out data/sample.tif
# Simulate a synthetic drone flight over the basemap
vps simulate --basemap data/sample.tif --path data/flights/loop.yaml --out runs/loop
# Run the localization engine over a flight bundle and score it
vps run runs/loop --basemap data/sample.tif --out results/loop --matcher orb
# Full end-to-end demo in one shot
vps demo --out demo_out
# Offline desktop app — load a basemap + photo + altitude → GPS (no browser)
vps gui

On Windows, prefix with .\.venv\Scripts\ or activate the venv first:

.\.venv\Scripts\Activate.ps1
vps demo --out demo_out

Web app — image + altitude → GPS

For hands-on testing against real satellite imagery, install the app extra and launch the browser app.

.\.venv\Scripts\python.exe-m pip install -e ".[app]"
.\.venv\Scripts\vps.exe app

In the app you: (1) pick an area (search a place name or enter lat/lon) and fetch satellite imagery — it's downloaded, reprojected to UTM, and cached locally so the app works offline afterwards; (2) upload a photo of that area or click "Generate test image" to crop one from the imagery (with a known true location); (3) enter the altitude in metres and hit Localize. You get the GPS coordinate, a pin on the map, the recovered heading, and the feature-match evidence. For test images it also reports the localization error in metres.

Imagery source is selectable: Google Satellite (most current; personal-use fetch — its ToS forbids redistribution) or Esri World Imagery (free, usable in offline/shippable builds).

Single-image localization is global (prior-free): the engine searches the whole fetched area to find where the image belongs. Because you fetch a specific area rather than the whole planet, the search is fast and reliable.

Desktop app — fully offline

The native Tkinter desktop app requires no browser, no server, and no internet connection at all. Load a local basemap GeoTIFF, drop in a photo, enter the altitude, and click Localize:

.\.venv\Scripts\vps.exe gui

Or double-click Localize.cmd at the repo root (Windows only).

The success flag

Every PositionEstimate returned by the engine carries a success: bool field. This is the per-fix trustworthiness verdict for a downstream consumer like ArduPilot/MAVLink: inject the GPS fix only when success is True.

success=True requires all of:

• At least 12 RANSAC inliers (robust geometric agreement between frame and map)
• Scale factor within 20% of 1.0 (no wildly wrong match scale)
• Prediction error below 25 m vs. the motion filter (streaming path only)

Dead-reckoned frames and failed matches always have success=False. The localized field on the same object uses a looser 80 m gate and is retained for diagnostic purposes.

Outputs

vps run writes three files to the output directory:

• track.geojson — estimated and ground-truth flight paths plus per-frame diagnostic points. Drag it onto geojson.io or into QGIS to see the estimated track overlaid on a map. Each point includes a success property.
• errors.csv — one row per frame: estimated/true position, error in metres, heading error, inlier count, confidence, success flag, prediction error.
• report.json — the accuracy summary (localized %, trusted %, mean/median/ p95/max position error, heading error).

Bring your own basemap & flight

• Basemap: any georeferenced GeoTIFF in a projected CRS works in place of the synthetic one — pass it to --basemap.
• Flight: edit data/flights/loop.yaml (waypoints, altitude, speed, camera FOV, and the degradation/telemetry-noise model) and pass it to simulate.

A note on accuracy

The synthetic camera frames are rendered from the same basemap they are matched against, then degraded (lighting/season shift, sensor noise, motion blur, JPEG compression, noisy telemetry). The ~1 m median error reflects that controlled appearance gap. Real accuracy depends primarily on how much the live camera's appearance differs from the stored reference map (season, time of day, sensor type). Closing that gap is what the deep-learning matcher on the roadmap is for.

Deep-learning matcher (optional)

Two deep-learning backends ship behind the Matcher seam: LoFTR and DINOv2 (matchers/deep.py, matchers/dino.py). They require PyTorch:

# Install PyTorch first with your CUDA version — see pytorch.org/get-started
pip install torch --index-url https://download.pytorch.org/whl/cu121
# Then install the deep extra
.\.venv\Scripts\python.exe-m pip install -e ".[deep]"

Select via --matcher loftr or --matcher dino.

Current status: off-the-shelf pretrained weights do not beat ORB for this task. LoFTR was trained on ground-level photos (MegaDepth) and is out-of-distribution on top-down aerial imagery. DINOv2 works for same-modality matching but fails genuine cross-modal (satellite tiles ↔ live camera). Achieving competitive cross-modal results requires training on UAV↔satellite pairs (UAV-VisLoc/AerialVL dataset), which is the remaining planned work. The default matcher stays ORB and is recommended for all current use.

Testing

# Windows
.\.venv\Scripts\python.exe-m pytest
# macOS / Linux
.venv/bin/python -m pytest

69 tests pass, 9 are skipped (8 Streamlit tests that activate after pip install -e ".[app]", 1 deep-matcher test). Zero failures.

Test file	What it checks
test_e2e.py	Full pipeline — synthetic flight, median error < 3 m
test_geo.py	CRS / affine math, pixel↔world round-trip
test_rectify.py	Simulator ↔ rectifier are exact inverses (correlation ≥ 0.995)
test_matcher.py	ORB/SIFT match result structure and inlier count
test_localize.py	Prior-free global localization on a synthetic basemap
test_success_flag.py	success flag and prediction_error() on good/bad frames
test_report.py	DataFrame construction, summarize, GeoJSON, success columns
test_tiles.py	Tile math, cache hit/miss, mocked network calls
test_gui.py	Desktop-app helper functions; Streamlit helpers (skip without [app])
test_app.py	Streamlit app helpers (skipped without [app])

Project layout

src/visual_positioning/
types.py # shared dataclasses (Frame, Telemetry, PositionEstimate, …)
geo.py # CRS / affine helpers (pixel ↔ world ↔ lat/lon)
basemap.py # georeferenced raster: load/save, synthesize, patch extraction
rectify.py # pose → rotation, ground↔image homography, orthorectification
matchers/ # Matcher protocol + ORB/SIFT + deep backends (LoFTR, DINOv2)
fusion.py # constant-velocity motion filter + gating + prediction_error
positioning.py # Estimator — estimate() (flight) + localize() (single image)
report.py # accuracy metrics + GeoJSON/CSV/JSON outputs
tiles.py # satellite tile fetch → georeferenced basemap (Google/Esri)
visualize.py # shared match/track overlays (used by app + viz script)
sim/ # synthetic flight simulator + flight-bundle I/O
cli/ # `vps` command-line front-end
app/ # Streamlit web app (vps app) + Tkinter desktop app (vps gui)
data/flights/ # flight specs (YAML)
scripts/visualize.py # render track + match-evidence PNGs from a run
tests/ # full test suite

Roadmap

Done:

• Core engine — Estimator.estimate (orthorectify → match → geo-solve → fuse)
• Classical matcher (ORB/SIFT) behind the Matcher protocol
• Synthetic flight simulator + flight-bundle I/O
• CLI (vps make-basemap/simulate/run/demo) + accuracy reporting
• Temporal fusion (constant-velocity filter + gating)
• Test suite (geo, rectify, matcher, end-to-end, localize, success, report, tiles, gui)
• Single-image localization — Estimator.localize (prior-free global search)
• Coarse-to-fine global search — slides cheap overlapping coarse windows then refines the best candidate at native resolution (~11 s, sub-metre, 0 misses on a 120 m-altitude photo on a 5 km map)
• Satellite tile fetch → georeferenced basemap (tiles.py: Google + Esri, reprojected to UTM, cached → offline after first fetch)
• Streamlit web app (vps app): fetch satellite area → image + altitude → GPS
• Offline desktop app (vps gui): native Tkinter, no browser/server/network
• Deep-learning matcher infrastructure — LoFTR and DINOv2 backends behind the Matcher seam, GPU-accelerated, weights cached locally. Off-the-shelf weights do not yet beat ORB (see note above); training is the remaining step.
• Failure/uncertainty detection — success: bool on PositionEstimate; the per-fix trustworthiness verdict for ArduPilot/MAVLink

Planned:

• Free fully-offline imagery tier (Esri/USGS bundled cache) for field builds
• ArduPilot / MAVLink link (live attitude + altitude in, pose out) — final goal
• Trained cross-modal deep matcher — DINOv2 + structure-constrained matching trained on UAV↔satellite pairs (UAV-VisLoc/AerialVL); this is the remaining work to make the deep backend competitive with ORB
• Learned image-retrieval coarse stage — precompute foundation-model descriptors for basemap tiles offline; retrieve candidate regions by descriptor similarity instead of brute-force sweep (scalable for large maps)
• Multi-altitude / multi-scale robustness — multi-resolution features to handle scale variation from altitude changes
• Real recorded-flight ingestion / live camera