street watch

In the dog house. The mechanism is real (Spike 0: a camera's own frames sit 26x tighter in embedding space than across cameras), so the wall genuinely lights up when a scene changes. But scene-change does not predict TfL road incidents: on a fair out-of-sample test with full banks, anomaly-vs-incident AUROC is 0.54 and average precision is 1.09x the base rate. The weak label may be the wrong oracle, but no signal was shown against it. Kept public because the engineering is real.

Unsupervised, factory-style visual anomaly detection over Transport for London's 882 live JamCams, validated against TfL's own road-incident feed. Each camera learns its own normal from CLIP scene embeddings and optical-flow motion; a live frame is scored by how far it drifts from that normal. The screen is the product: a Watch_Dogs wall of looping feeds, tiles glow and rise with a targeting box on the action when a feed is unusual, click one to fly to it with the nearest incident.

No GPU: CPU ONNX end to end (fastembed CLIP, OpenCV optical flow). No paid LLM: the "ask the wall" box is CLIP text-image search.

Architecture (FTI on Hopsworks)

flowchart LR
J[TfL JamCams<br/>882 live feeds] --> F1[F1 fetch-frames<br/>CLIP embed + motion + box]
D[TfL Road Disruptions] --> F2[F2 fetch-incidents<br/>weak labels]
F1 --> CF[(camera_frames FG<br/>online + offline)]
F1 --> CM[(camera_meta FG)]
F2 --> IN[(incidents FG)]
CF --> T1[T1 train-normalcy<br/>per-camera memory bank]
IN --> T1
T1 --> M[[street_normalcy<br/>model registry]]
CF --> APP[the wall<br/>in-app scoring, live video]
CM --> APP
IN --> APP
M --> APP

Loading

• F1 fetch-frames (every 15 min): polls all 882 cameras, embeds each frame (CLIP, CPU), computes optical-flow motion and the motion bounding box, writes camera_frames (online store keeps the latest per camera; offline accumulates history). Shared no-skew street_features.py.
• F2 fetch-incidents (every 15 min): road disruptions into incidents, the weak validation labels.
• T1 train-normalcy (every 12h): fits a global PCA, builds each camera's normal (a memory bank of its frames plus robust motion baselines), scores the anomaly against incidents as weak labels, registers street_normalcy.
• the wall (app/server.py): loads the champion, scores the online frames in-process (the model is small, no KServe), ranks the feeds, and serves the live grid. Hot-reloads the champion hourly so retrains reach the wall automatically.

The model

Per camera: anomaly = max(scene novelty, motion deviation). Scene novelty is the distance from the live frame to the k nearest normal frames in that camera's memory bank (PatchCore idea on street scenes). Motion deviation is the robust z of the optical-flow motion against the camera's own baseline. A reason in plain words comes with every flag ("traffic quieter/stopped", "the scene does not match this camera's usual view").

Validation is the honest gate, and this build did not clear it. Out-of-sample (frames scored against a champion that never saw them, banks full at 64 frames per camera), anomaly-vs-incident AUROC is 0.536 and average precision is 1.09x the base rate, motion-only baseline 0.462. See eval_oos.py. The per-camera novelty is real, but it does not track TfL road disruptions. The weak label (a disruption within 300m in a time window) may be the wrong oracle, since a logged disruption is often not visible in the frame, but no signal was shown against it. That is why it sits in the dog house.

Run it

python deploy_env.py # CPU vision env (fastembed, opencv)
python deploy_frames.py && python deploy_incidents.py # F1 + F2 jobs
hops job schedule fetch-frames "0 0/15 * * * ?" # every 15 min
hops job schedule fetch-incidents "0 7/15 * * * ?"
python deploy_train.py # T1 job
hops job schedule train-normalcy "0 30 3/12 * * ?" # every 12h
python deploy_appenv.py && python app/deploy_app.py # the wall

Built on Hopsworks. Feeds © Transport for London.