Skip to content

Feature keypoints double-scaled for frames larger than featureImageMinSize (marker mis-localized at ~2x position) #43

Description

@kalwalt

Summary

WebARKitTracker mis-localizes the marker for any frame larger than featureImageMinSize (640x480): the homography and solvePnP pose place tracked content at roughly 2x the marker's true position. Typical 640x480 webcams are unaffected, which is why this has gone unnoticed.

Root cause

WebARKitTrackerImpl::initialize() computes _featureDetectScaleFactor from an image-pyramid downsample level (_featureDetectPyrLevel), intended to scale keypoints detected on a downsampled image back up to full-frame coordinates.

But the downsampling is disabled — the pyrDown block in resetTracking() is commented out and extractFeatures() runs on the full-resolution frame. The matched frame keypoints (already full-frame) are then still multiplied by the factor in MatchFeatures():

finalMatched1[i].pt.x *= _featureDetectScaleFactor[0];
finalMatched1[i].pt.y *= _featureDetectScaleFactor[1];

With featureImageMinSize = 640x480:

Frame pyrLevel factor effect
640x480 0 1.0 OK (hidden)
2000x1500 1 2.0 every keypoint doubled
1920x1080 1 2.0 broken

Doubled keypoints -> homography fit to doubled points (degenerate/exploded bbox) -> solvePnP fed doubled image points -> wrong translation -> content rendered at ~2x position.

Evidence

Centroid of matched reference vs frame keypoints on a 2000x1500 frame: ref(1034,908) -> frame(1816,1250), while the marker physically sits at ~(908,625). (1816,1250) = 2 x (908,625).

Fix

While detection runs full-res, the factor must be identity:

_featureDetectScaleFactor = cv::Vec2f(1.0f, 1.0f);

(Keep the pyramid-derived computation commented out; restore it together with the pyrDown detection path if downsampled detection is re-enabled.)

Follow-up

The disabled downsampling block (// This if cond. doesn't works as expected in artoolkitx ...) and the related _featureDetectScaleFactor machinery should be reconciled with the ArtoolkitX OCVT reference: either restore downsampled detection (perf) with a consistent rescale on both the match side and createFeatureMask, or remove the now-vestigial scale path entirely.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions