Skip to content

Add live-webcam Teblid AR example + examples index (#36) #37

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
kalwalt merged 2 commits into from
Jun 12, 2026
Merged

Add live-webcam Teblid AR example + examples index (#36) #37

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
10b75e1
feat(#36): add live-webcam Teblid AR example + examples index
kalwalt Jun 10, 2026
28ed6f9
docs(example): note the tracking-loss limitation in the webcam example
kalwalt Jun 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
129 changes: 129 additions & 0 deletions docs/design-webcam-teblid-example.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,129 @@
# Design: Live-webcam Teblid AR example (issue #36)

Branch: `feat/teblid-webcam-example` (webarkit/webarkit-testing). Adds a new
working live-camera AR example using the Teblid tracker, plus an `index.html`
landing page for all examples. **No existing example is modified or removed**
(the cleanup half of #36 is deferred).

Depends on the library fixes merged in webarkit/WebARKitLib#45 (#43 scale-factor,
#35 projection X-sign, #42 D*R*D handedness) — so the example consumes
`matrixGL_RH` directly with no render-side pose correction.

## Understanding Summary

- **What:** a new live-webcam AR example using the **Teblid** tracker, *added*
to `examples/` (nothing removed), plus a new `examples/index.html` listing all
examples.
- **Why:** the existing webcam examples are broken; we need a working live
reference built on the now-correct library.
- **Who:** WebARKit developers wanting a live-camera Teblid starting point.
- **Shape:** the static-image example's *conventions* (consume `matrixGL_RH`
directly, projection as-is, no pose hack) + the bufferCopy example's
*continuous main↔worker ping-pong loop* + a `getUserMedia` feed.
- **Hard constraints:** don't touch/delete the other examples; RGBA end-to-end
(the tracker converts via `convert2Grayscale`); **never mirror the processed
feed**; no library changes and no rebuild (pure example HTML/JS).

## Assumptions

- **Files:** `threejs_teblid_webcam_ES6_example.html`,
`threejs_teblid_webcam_worker_ES6.js` (the `start()` module),
`worker_teblid_webcam_threejs.js` (dedicated worker), `index.html`.
- **Marker:** `data/pinball.jpg`, decoded to RGBA via the static example's
Image → canvas → `getImageData` → `loadTrackerGrayImage` path.
- **Content:** cube (0.6) at +Z 0.3 **+ `AxesHelper`**, attached to `root`,
anchored at the marker origin (object 0,0,0 = reference top-left).
- **Capture:** 640×480, `facingMode:"environment"`, reusing `initCamera.js`;
capture size taken from `video.videoWidth/Height` (actual, not requested);
`CAPTURE_W`/`CAPTURE_H` constants at the top of the HTML.
- **No build artifacts, no library edits.** Uses the existing `dist/WebARKit.js`.

## Decision Log

| Decision | Alternatives | Why |
|---|---|---|
| **bufferCopy ping-pong** (recycle one transferable `ArrayBuffer`) | getImageData one-way per frame; SharedArrayBuffer zero-copy | Continuous video needs to avoid per-frame allocation/GC; SAB needs COOP/COEP headers (out of scope). The bufferCopy example exists for exactly this. |
| **New dedicated `worker_teblid_webcam_threejs.js`** emitting `matrixGL_RH` | Modify `worker_bufferCopy_threejs.js`; reuse static `worker_threejs.js` | Isolation honors "don't touch other examples"; the static worker can't ping-pong (one-way transfer). One duplicated ~80-line worker is acceptable. |
| **Match-aspect display, projection as-is** | Letterbox-pad + `ratioW/ratioH` projection scaling (bufferCopy style); CSS cover, no compensation | Same proven convention as the corrected static example; no projection arithmetic to get wrong; exact AR alignment. Letterbox bars on mismatched windows are acceptable. |
| **640×480 `environment`, reuse `initCamera.js`** | 1280×720 default; `user` (front) default | Fast, lands on the factor-1.0 path, ideal for pointing at a printout; front feeds invite mirroring which would break handedness. Resolution is a constant so the "also works at 1280×720" criterion is one edit away. |
| **Consume `matrixGL_RH` directly, no correction** | Re-add an example-side pose tweak | The library now emits a correct D*R*D pose + standard projection (#45). |
| **Hide content on `not found`** (`world = null`) | Freeze last pose (static example behavior) | Live tracking: content should disappear when the marker leaves frame. |
| **Use actual `video.videoWidth/Height`** | Trust the requested 640×480 | Browsers may not honor the exact request; using the real size keeps tracker init + projection consistent. |
| **`index.html`: grouped, relative links, "under review (#36)" group** | Flat alphabetical; delete legacy now | Honest about state without removing anything pre-audit; relative links work from `examples/` on any static server. |
| **cube + axes** | cube-only | Axes give an immediate, unambiguous read of the marker frame (red→right, green→up, blue→toward viewer). |

## Final Design

### Files (4 new; nothing else touched)

1. **`examples/threejs_teblid_webcam_ES6_example.html`** — `<video id="video">`
(behind, un-mirrored) + `<canvas id="canvas">` overlay; stats + loader
(reused pattern); `CAPTURE_W`/`CAPTURE_H` constants. On `load`:
`initCamera(CAPTURE_W, CAPTURE_H)` → read `video.videoWidth/Height` →
`start('./data/pinball.jpg', video, w, h, …)`.
2. **`examples/threejs_teblid_webcam_worker_ES6.js`** — `start()` module:
three.js scene/camera/`root` with cube + `AxesHelper`; spawns the worker;
decodes `pinball.jpg` → RGBA → `initTracker`; bufferCopy `process()`;
`found()` consumes `matrixGL_RH`; `requestAnimationFrame` render loop.
3. **`examples/worker_teblid_webcam_threejs.js`** — dedicated worker:
`init_raw` → `loadTrackerGrayImage(RGBA)` → `initFrameBuffer(RGBA)` →
`getCameraProjectionMatrix`; `getMarker` → emit `matrixGL_RH`;
`process_raw(next, RGBA)`; **returns the buffer** (transfer back) for ping-pong.
4. **`examples/index.html`** — navigation page linking every example (grouped:
feature-tracker / three.js AR / under-review-#36).

### Per-frame data flow

```
video → drawImage → getImageData → set() into recycled ArrayBuffer
→ worker.postMessage(transfer buffer) → process_raw(RGBA)
→ tracker (convert2Grayscale internally) → getMarker(matrixGL_RH)
→ postMessage('found'|'not found', transfer buffer BACK) → main
→ found(): world = matrixGL_RH (or null); process() next frame
```
A separate `requestAnimationFrame` loop renders `root` from the latest `world`.
The ping-pong sets the processing cadence; the render loop is independent.

### Projection / display
- Capture size from `video.videoWidth/Height`.
- `renderer.setSize()` + `#canvas`/`#video` sized to the capture aspect
(letterboxed in-window). Projection set **once** on `loadedTracker`, used
**as-is** (no `ratioW/ratioH` scaling).

### Loop / lifecycle
- Kick `process()` after `loadedTracker`; re-`process()` on each worker reply.
- Gate `process()` on `video.readyState >= HAVE_CURRENT_DATA`.
- `not found` → `world = null` → content hidden.

### No-mirror guarantee
- `drawImage(video,…)` with no flip; no `scaleX(-1)` on the processed canvas or
the displayed `<video>`.

### Error handling
- `getUserMedia` rejected → message in loader, log, stop.
- Worker `init_raw` error → `console.error`, loader stays.

## Non-functional notes
- **Performance:** detection runs full-res every frame (the downsample/guard is
deferred to webarkit/WebARKitLib#44); 640×480 is the smooth default, 1280×720
is the "higher-res" check. No hard fps SLA (reference demo).
- **Scope:** single marker, single example; the examples-cleanup half of #36 is
deferred (legacy examples only get labelled in `index.html`).

## Risks
- **Aspect mismatch** between camera capture and window → letterbox bars
(accepted) — but if `video` and `#canvas` ever diverge in size/position, AR
drifts. Mitigation: size both from the same capture dimensions in one place.
- **Front-camera mirroring** if someone switches to `facingMode:"user"` and adds
a selfie flip → re-introduces a handedness flip. Mitigation: documented "never
mirror the processed feed".
- **Browser not honoring requested resolution** → handled by reading actual
`videoWidth/Height`.
- **getCorners/debug** not included (kept clean); re-add the static example's
debug overlay temporarily if localization needs verifying.

## Acceptance (from #36)
- [ ] New webcam Teblid example tracks the pinball marker live, content correctly
localized + right-handed (red→right, green→up, blue→toward viewer).
- [ ] Works at 640×480 and at least one higher resolution (e.g. 1280×720).
- [ ] No existing example modified/removed; `index.html` lists all examples.
88 changes: 88 additions & 0 deletions examples/index.html
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>WebARKit Examples</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
<style>
:root { color-scheme: light dark; }
body {
font-family: system-ui, -apple-system, "Segoe UI", Roboto, sans-serif;
max-width: 820px;
margin: 0 auto;
padding: 2rem 1.25rem 4rem;
line-height: 1.5;
}
h1 { margin-bottom: 0.25rem; }
.sub { color: #777; margin-top: 0; }
h2 {
margin-top: 2rem;
padding-bottom: 0.35rem;
border-bottom: 1px solid rgba(127, 127, 127, 0.35);
font-size: 1.15rem;
}
ul { list-style: none; padding: 0; }
li { margin: 0.55rem 0; }
a { font-weight: 600; text-decoration: none; color: #2563eb; }
a:hover { text-decoration: underline; }
.desc { color: #777; font-weight: 400; }
.note { font-size: 0.85rem; color: #777; }
.badge {
display: inline-block;
font-size: 0.7rem;
font-weight: 700;
padding: 0.05rem 0.4rem;
border-radius: 4px;
background: #16a34a;
color: #fff;
vertical-align: middle;
margin-left: 0.4rem;
}
.badge.review { background: #d97706; }
</style>
</head>
<body>
<h1>WebARKit Examples</h1>
<p class="sub">Browser AR examples for the WebARKit tracking library.</p>

<h2>Feature-tracker examples</h2>
<ul>
<li><a href="./teblid_example.html">Teblid</a>
<span class="desc">— TEBLID descriptor tracker.</span></li>
<li><a href="./akaze_example.html">AKAZE</a>
<span class="desc">— AKAZE feature tracker.</span></li>
<li><a href="./orb_example.html">ORB</a>
<span class="desc">— ORB feature tracker.</span></li>
<li><a href="./freak_example.html">FREAK</a>
<span class="desc">— FREAK descriptor tracker.</span></li>
</ul>

<h2>three.js AR examples</h2>
<ul>
<li><a href="./threejs_teblid_static_image_ES6_example.html">Teblid &mdash; static image</a>
<span class="badge">working</span>
<span class="desc">— AR overlay on a still image, Teblid tracker.</span></li>
<li><a href="./threejs_teblid_webcam_ES6_example.html">Teblid &mdash; webcam</a>
<span class="badge">new</span>
<span class="desc">— live camera AR, Teblid tracker (bufferCopy ping-pong).</span></li>
</ul>

<h2>Other / under review <span class="note">(see issue #36)</span></h2>
<p class="note">These older examples are kept for now but may be broken or
superseded; they are being audited in issue #36.</p>
<ul>
<li><a href="./teblid_example_4_threejs.html">Teblid + three.js</a>
<span class="badge review">review</span></li>
<li><a href="./teblid_advanced_example.html">Teblid advanced</a>
<span class="badge review">review</span></li>
<li><a href="./threejs_speedy_ES6_example.html">three.js + Speedy</a>
<span class="badge review">review</span></li>
<li><a href="./threejs_bufferCopy_ES6_example.html">three.js bufferCopy</a>
<span class="badge review">review</span></li>
<li><a href="./threejs_ES6_example.html">three.js ES6</a>
<span class="badge review">review</span></li>
<li><a href="./threejs_ES6_jsfeatNext_example.html">three.js + jsfeatNext</a>
<span class="badge review">review</span></li>
</ul>
</body>
</html>
124 changes: 124 additions & 0 deletions examples/threejs_teblid_webcam_ES6_example.html
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,124 @@
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<title>WebARKit Teblid - Webcam ES6 example</title>
<meta name="viewport" content="width=device-width, initial-scale=1, minimum-scale=0.5, maximum-scale=1">
<link rel="stylesheet" href="css/nft-style.css">
<style>
/* On-screen tracking status (console logging is kept as well). */
#status {
left: 50%;
bottom: 14px;
transform: translateX(-50%);
margin: 0;
z-index: 200;
padding: 0.4rem 0.9rem;
font-size: 0.9rem;
font-weight: 700;
color: #b91c1c; /* searching: red */
}
#status.tracked { color: #15803d; } /* tracked: green */
</style>
</head>
<body>

<div id="loading">
<img src="data/aframe-k.png"/>
<span class="loading-text">Loading, please wait</span>
</div>

<!--
==================
STATS
==================
-->
<div id="stats" class="ui stats">
<div id="stats1" class="stats-item">
<p class="stats-item-title">Main</p>
</div>
<div id="stats2" class="stats-item">
<p class="stats-item-title">Worker</p>
</div>
</div>

<!-- On-screen tracking status -->
<div id="status" class="ui">Searching for marker&hellip;</div>

<div id="app">
<!--
Live camera feed. #video and #canvas are both full-bleed object-fit:cover
(nft-style.css). Since the canvas drawing buffer is sized to the capture
resolution (same aspect as the video), the cover-crop is identical on both,
so the AR overlay aligns with the video. The video is NOT mirrored.
-->
<video id="video" autoplay muted playsinline></video>
<canvas id="canvas"></canvas>
</div>

<script src="js/stats.min.js"></script>
<script src="js/three.min.js"></script>
<script src="initCamera.js"></script>

<script>
function setTrackerType() {
return 'teblid';
}
</script>

<script src="threejs_teblid_webcam_worker_ES6.js"></script>

<script>
/**
* STATS
*/
var statsMain = new Stats();
statsMain.showPanel(0);
document.getElementById('stats1').appendChild(statsMain.dom);

var statsWorker = new Stats();
statsWorker.showPanel(0);
document.getElementById('stats2').appendChild(statsWorker.dom);

// Capture resolution. Change to 1280 x 720 to verify the higher-res path.
const CAPTURE_W = 640;
const CAPTURE_H = 480;

window.addEventListener('load', async () => {
console.log('init WebARKit Teblid Webcam...');

try {
const video = await initCamera(CAPTURE_W, CAPTURE_H);
// Use the ACTUAL capture size (browsers may not honor the request).
const vw = video.videoWidth || CAPTURE_W;
const vh = video.videoHeight || CAPTURE_H;
console.log('camera size:', vw, 'x', vh);

initTargetCanvas(vw, vh);
start(
'./data/pinball.jpg',
video,
vw,
vh,
function () { statsMain.update(); },
function () { statsWorker.update(); }
);
} catch (e) {
console.error('Camera init failed:', e);
const loader = document.getElementById('loading');
if (loader) {
loader.querySelector('.loading-text').innerText =
'Camera access failed: ' + (e && e.message ? e.message : e);
}
}
});

function initTargetCanvas(width, height) {
const canvas = document.querySelector('#canvas');
canvas.width = width;
canvas.height = height;
}
</script>

</body>
</html>
Loading