ReverseProjection

reverse-projection.mp4

A live augmented-reality application that turns a webcam into a position sensor: hold your phone in front of the camera and an interactive map appears, perfectly aligned with your physical location. Touch the phone screen to draw annotations that appear on both the phone and the large display simultaneously.

Anyone who opens the phone page connects automatically.

---

How It Works

1. The phone displays a single ArUco fiducial marker (ID 0) centred at the top of the screen so the webcam can detect and track it.
2. The laptop accesses the webcam, detects the marker using jsartoolkit5, and derives the phone's position and rotation from the marker's four corner vertices.
3. Both devices communicate over WebSockets (Socket.io) through a local Node.js server running on your network.
4. The laptop map shows a bounding box representing the area currently visible on the phone's mini-map.
5. Touching the phone screen sends lat/lng coordinates back to the laptop and draws a stroke on both maps simultaneously.

---

Mapping Model

The project uses three different coordinate spaces. Keeping them separate is the key to making the overlays line up.

1. Camera space The laptop webcam sees the physical phone. Marker detection gives a phone centre (nx, ny) in normalised camera coordinates and a phone rotation θ.

2. Phone viewport space The phone reports the size of the actually visible interactive area, not the whole page. This is the rectangle below the marker and above the bottom status bar. That reported width and height are the only dimensions that may be used to infer the visible viewport.

3. Content space Each example maps the visible phone viewport into its own content space.

   • Map mode: geographic map bounds.
   • Screen mode: source video pixels from the captured desktop stream.
   • Whiteboard mode: whiteboard world coordinates.

The important rule is:

• The blue overlay on the laptop must represent the phone's visible content viewport in content space.
• It must not represent the physical phone body in camera space.

Screen mode mapping

Screen mode uses the exact same viewport transform on both devices.

1. The phone renders the incoming desktop stream into its visible area using a cover transform:
   • base scale = max(phoneW / sourceW, phoneH / sourceH)
   • total scale = base scale * zoom
2. The phone then translates the scaled source so that the tracked point (nx, ny) becomes the centre of the visible phone viewport.
3. That translation is clamped so the source never exposes empty edges.
4. The laptop overlay must compute the inverse of that same transform and draw the resulting source rectangle on top of the displayed preview.
5. If rotation is enabled, the same viewport rectangle is rotated by the tracked phone angle. If rotation is disabled, both the phone and the overlay stay axis-aligned.

This means the laptop overlay for Screen mode must be derived from:

• source video size
• reported visible phone viewport size
• zoom
• tracked (nx, ny)
• tracked rotation when enabled

It must not be derived from the marker's apparent size in the webcam image.

Map mode mapping

Map mode uses the reported visible phone viewport size to convert the tracked phone centre into a geographic rectangle.

1. The laptop maps (nx, ny) into the current map bounds.
2. The reported phone viewport width and height are converted to fractions of the camera frame using the detected physical scale.
3. Those fractions are then converted into fractions of the current Leaflet map bounds.
4. If rotation is enabled, the rectangle is rotated. Otherwise it stays upright.

The same visible viewport dimensions must be used consistently on the phone, in the laptop state, and in the laptop overlay.

---

Getting Started

Step 1 — Install Node.js

Node.js is the JavaScript runtime the server needs. Install it once; it stays on your machine.

macOS

The easiest way is Homebrew, a package manager for macOS. If you have never used it:

1. Open Terminal (press Command+Space, type "Terminal", press Enter).

2. Paste the following command and press Enter. It will ask for your Mac password:

   /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
   

3. After Homebrew finishes, install Node.js:

   brew install node
   

4. Verify the installation:

   node --version
   

   You should see something like v20.x.x.

Windows

1. Open a web browser and go to https://nodejs.org.

2. Download the LTS installer (the button labelled "LTS" — Long Term Support).

3. Run the downloaded .msi file and follow the installer steps. Leave all options at their defaults.

4. When it finishes, open Command Prompt (press Windows+R, type cmd, press Enter) and verify:

   node --version
   

Linux (Ubuntu / Debian)

sudo apt update
sudo apt install nodejs npm
node --version

For other distributions, see https://nodejs.org/en/download/package-manager.

---

Step 2 — Download the project

If you have Git installed:

git clone https://github.com/lemio/ReverseProjection.git
cd ReverseProjection

If you do not have Git, download the ZIP from GitHub (click the green "Code" button, then "Download ZIP"), unzip it, and open a terminal inside the folder.

---

Step 3 — Install dependencies and start the server

In your terminal, inside the project folder:

npm install
npm start

You should see:

ReverseProjection server running at http://localhost:3000

Leave this terminal window open — the server must keep running.

---

Step 4 — Open the laptop app

Open a browser (Chrome or Edge recommended for best webcam support) and go to:

http://localhost:3000

Allow camera access when the browser asks. The webcam feed will appear.

---

Step 5 — Connect a phone

The phone must be on the same Wi-Fi network as the laptop.

1. On the laptop, click Copy Phone Link in the toolbar. This copies the URL to your clipboard.
2. Open that URL on the phone's browser, or click Show QR Code and scan it with the phone camera.
3. The phone will connect automatically and display an interactive map.

To find your laptop's local IP address (for typing the URL manually):

• macOS / Linux: run ifconfig | grep "inet " in Terminal — look for a number like 192.168.x.x
• Windows: run ipconfig in Command Prompt — look for "IPv4 Address"

Then open http://192.168.x.x:3000/phone on the phone.

---

Using the Application

• Hold the phone face-up in front of the webcam. A single black-and-white marker centred at the top of the screen allows the webcam to track position and rotation.
• The laptop map shows a blue rectangle representing the area currently visible on the phone.
• Touch and drag on the phone to draw annotations. They appear on both screens simultaneously.
• Use the toolbar buttons to invert controls or enable map rotation.

Toolbar controls

Button	Description
Map	Activates the map example (the only built-in example)
Invert	Flips the phone's position mapping so moving up moves the map north
No Rotation	When toggled to "Rotating", the phone's yaw rotates the mini-map
Copy Phone Link	Copies the LAN phone URL to the clipboard
Show QR Code	Displays a QR code for the LAN phone URL

Lighting tips

• Keep the phone screen brightness high — the marker needs clear contrast.
• Avoid direct glare on the phone screen.
• If detection is unreliable, reduce ambient light reflections.
• When the phone is lost, the marker automatically grows to help re-acquisition.

---

File Structure

server.js Node.js / Express / Socket.io server
public/
index.html Laptop UI (webcam + overlay + map panel)
css/style.css Dark professional theme
js/
app.js Main orchestrator (webcam loop, detection, state)
jsarDetector.js Detects marker ID 0 via jsartoolkit5 — position and rotation from corners
homography.js Perspective-transform math (DLT algorithm)
vendor/
artoolkit.min.js jsartoolkit5 self-contained bundle
examples/
mapExample.js Leaflet map — phone position to geographic coordinate
phone/
index.html Phone PWA (auto-connects, no room code needed)
manifest.json
sw.js Service worker (offline cache)
css/phone.css
js/
phoneApp.js Auto-connection and example lifecycle
drawMarker.js Renders ArUco marker patterns onto canvas
examples/
mapPhone.js Leaflet mini-map tracking the phone's geographic position

---

Technical Notes

• Detection uses jsartoolkit5 with a single 3x3 barcode marker (ID 0) centred at the top of the phone screen. The marker's four corners provide position and rotation.
• The server detects the machine's LAN IP at startup and exposes it via /api/config so the laptop app can generate a correct phone URL for the QR code and copy-link button.
• All devices share a single server session — no room codes or pairing required.
• The phone mini-map renders at three zoom levels deeper than the laptop map and freezes during active drawing to keep strokes clean.
• Drawn paths are placed in a dedicated Leaflet pane (drawPane) at z-index 650 with overflow: visible to prevent clipping at tile boundaries.