Skip to content

Fix AR pose/localization on >640x480 frames: keypoint scale bug, projection X-sign, marker handedness - #45

Merged
kalwalt merged 3 commits into
webarkit:devfrom
kalwalt:fix/pose-drd-convention
Jun 10, 2026
Merged

Fix AR pose/localization on >640x480 frames: keypoint scale bug, projection X-sign, marker handedness#45
kalwalt merged 3 commits into
webarkit:devfrom
kalwalt:fix/pose-drd-convention

Conversation

@kalwalt

@kalwalt kalwalt commented Jun 9, 2026

Copy link
Copy Markdown
Member

Summary

Fixes AR pose/localization for the OCVT planar tracker on frames larger than featureImageMinSize (640x480) — the case exercised by the webarkit-testing static-image Teblid example at 2000x1500. What presented as a single "mirror" was three orthogonal bugs, headlined by a feature-detection scale bug that masked the others:

  1. Keypoint scale-factor double-applicationroot cause (Feature keypoints double-scaled for frames larger than featureImageMinSize (marker mis-localized at ~2x position) #43)
  2. Projection X-focal sign (Asymmetric -2*f_x/screenWidth in cameraProjectionMatrix — origin of X-mirror unclear #35)
  3. Marker-frame handedness (Restore CV->GL column negation in updateTrackable (D*R*D) to fix 3D pose orientation #42)

Plus a behavior-preserving refactor removing the now-dead scale machinery (#44).

Details

1. Scale-factor double-application (WebARKitTracker.cpp) — Fixes #43

Detection runs on the full-resolution frame (the pyrDown block in resetTracking() is disabled), but MatchFeatures still multiplied matched keypoints by _featureDetectScaleFactor (derived from the pyramid level):

Frame pyrLevel factor effect
640x480 (webcam) 0 1.0 OK — why it stayed hidden
2000x1500 1 2.0 every keypoint doubled

Doubled keypoints → marker localized at ~2x position → exploded homography → wrong solvePnP translation. Evidence: matched centroid ref(1034,908) → frame(1816,1250) = exactly 2×(908,625). Fix: force the factor to identity while detection is full-res.

2. Projection X-focal sign (WebARKitGL.cpp) — Refs #35

cameraProjectionMatrix used -2*f_x/w, mirroring screen-X. The modelview already applies the CV→GL Y,Z row negation (arglCameraViewRHf), so the projection must be plain pinhole with both focals positive (+2*f_x/w). Verified by projecting the tracked origin to NDC. gtests updated ([0]+1.7851850084276433).

3. Marker handedness — DRD (WebARKitPattern.cpp) — Refs #42

Negate the Y,Z rotation columns of trans (matching ArtoolkitX ARTrackable2d::updateWithTwoDResults). With the downstream row negation this yields D*R*D — a right-handed frame (X=right, Y=up, Z=toward viewer). The translation column is untouched, so position is unchanged.

4. Refactor (WebARKitTracker.cpp) — Refs #44

Remove the dead _featureDetectPyrLevel/_featureDetectScaleFactor members, the commented pyrDown/detection-guard block, and the no-op rescales. Restoring downsampled detection + the guard for performance is tracked in #44.

Risk / behavior

Testing

  • gtests updated for the projection sign.
  • Verified end-to-end via the static-image Teblid example (2000x1500): marker correctly localized, cube/axes anchored on the marker, right-handed (red→right, green→up, blue→toward viewer).

Review checklist

Companion example/build PR: webarkit/webarkit-testing (linked below).

🤖 Generated with Claude Code

kalwalt and others added 3 commits June 9, 2026 20:15
… full-res

Feature detection runs on the full-resolution frame (the pyrDown downsampling block in resetTracking() is disabled), but MatchFeatures still multiplied the matched frame keypoints by _featureDetectScaleFactor, derived from the image-pyramid level. For frames larger than featureImageMinSize (640x480) that factor is >= 2, so every matched keypoint was doubled -- the marker localized at ~2x its true position, exploding the homography and the solvePnP pose. 640x480 webcams have pyrLevel 0 (factor 1.0), which is why this stayed hidden.

Force _featureDetectScaleFactor to identity (1,1) while detection is full-res; keep the pyramid-derived computation commented out so it can be restored together with downsampled detection.

Fixes webarkit#43

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…edness

cameraProjectionMatrix: use +2*f_x/w (was -2*f_x/w). The modelview already applies the CV->GL Y,Z row negation (arglCameraViewRHf), so the projection must be the plain pinhole form with both focals positive; the negative X focal mirrored screen-X.

updateTrackable: negate the Y,Z rotation columns (D*R*D, matching ArtoolkitX ARTrackable2d::updateWithTwoDResults) so the marker frame is right-handed with Z toward the viewer. The translation column is untouched, so position is unchanged.

gtests: projection [0] -> +1.7851850084276433.

Refs webarkit#35, webarkit#42

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Feature detection has run on the full-resolution frame for some time (the pyrDown block and the single-marker detection guard were commented out), which made _featureDetectScaleFactor an identity no-op after the webarkit#43 fix. Remove the now-dead members (_featureDetectPyrLevel, _featureDetectScaleFactor), their initialize() computation, the commented pyrDown/guard block, the no-op keypoint rescale in MatchFeatures, and the no-op bbox divide in createFeatureMask.

Behavior is unchanged (all removed paths were x1/div1 or commented). Restoring downsampled detection + the detection guard for performance is tracked in webarkit#44.

Refs webarkit#43, webarkit#44

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@kalwalt

kalwalt commented Jun 9, 2026

Copy link
Copy Markdown
Member Author

Companion example + build PR: webarkit/webarkit-testing#35

@kalwalt kalwalt self-assigned this Jun 9, 2026
@kalwalt kalwalt added bug Something isn't working enhancement New feature or request C/C++ code concerning the C/C++ code design and improvements Emscripten labels Jun 9, 2026
@kalwalt kalwalt moved this from In progress to Review in progress in New markerless image tracking Jun 9, 2026
@kalwalt
kalwalt merged commit c1ae096 into webarkit:dev Jun 10, 2026
@github-project-automation github-project-automation Bot moved this from Review in progress to Done in New markerless image tracking Jun 10, 2026
kalwalt added a commit to webarkit/webarkit-testing that referenced this pull request Jun 10, 2026
webarkit/WebARKitLib#45 merged into dev (rebased to 1021616, f448ef3, c1ae096). Re-point the submodule from the fork branch commit to the canonical dev tip. Source is identical, so build/ and dist/ are unchanged.

Refs #30, webarkit/WebARKitLib#45

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
kalwalt added a commit to webarkit/webarkit-testing that referenced this pull request Jun 10, 2026
webarkit/WebARKitLib#45 merged into dev (rebased to 1021616, f448ef3, c1ae096). Re-point the submodule from the fork branch commit to the canonical dev tip. Source is identical, so build/ and dist/ are unchanged.

Refs #30, webarkit/WebARKitLib#45

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@kalwalt
kalwalt deleted the fix/pose-drd-convention branch June 15, 2026 22:07
kalwalt added a commit to webarkit/webarkit-testing that referenced this pull request Jun 21, 2026
webarkit/WebARKitLib#45 merged into dev (rebased to 1021616, f448ef3, c1ae096). Re-point the submodule from the fork branch commit to the canonical dev tip. Source is identical, so build/ and dist/ are unchanged.

Refs #30, webarkit/WebARKitLib#45

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working C/C++ code concerning the C/C++ code design and improvements Emscripten enhancement New feature or request

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

1 participant