Skip to content

Fix AR pose/localization on >640x480 frames: keypoint scale bug, projection X-sign, marker handedness #45

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
kalwalt merged 3 commits into from
Jun 10, 2026
Merged

Fix AR pose/localization on >640x480 frames: keypoint scale bug, projection X-sign, marker handedness #45

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
9eb49d9
fix(tracker): stop double-scaling matched keypoints when detection is…
kalwalt Jun 9, 2026
3a9c58a
fix(gl,pose): standard GL projection X sign + D*R*D marker-frame hand…
kalwalt Jun 9, 2026
fb78f85
refactor(tracker): remove vestigial pyramid-downsample/scale machinery
kalwalt Jun 9, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 9 additions & 1 deletion WebARKit/WebARKitGL.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,15 @@ void cameraProjectionMatrix(const std::array<double, 9>& calibration, double nea
double c_x = calibration.at(2); // Camera primary point x
double c_y = calibration.at(5); // Camera primary point y

projectionMatrix[0] = -2.0f * f_x / screenWidth;
// WebARKitLib#42: standard GL projection with BOTH focal terms positive.
// The modelview (matrixGL_RH) already performs the CV->GL handedness flip by
// negating the Y and Z rows in arglCameraViewRHf, so the projection itself must
// be the plain pinhole form. The previous negative X focal (-2*f_x/w) mirrored
// the marker horizontally: a marker origin at camera -X projected to +X (right)
// instead of -X (left). Verified against the tracked pose: origin (tx,ty,tz)=
// (-1.996, 2.223, -9.094) projects to NDC (-0.39, +0.58) (upper-left, on the
// marker) only with +2*f_x/w; the negative sign put it upper-right.
projectionMatrix[0] = 2.0f * f_x / screenWidth;
projectionMatrix[1] = 0.0f;
projectionMatrix[2] = 0.0f;
projectionMatrix[3] = 0.0f;
Expand Down
26 changes: 16 additions & 10 deletions WebARKit/WebARKitPattern.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,17 +55,23 @@ void WebARKitPatternTrackingInfo::getTrackablePose(cv::Mat& pose) {

void WebARKitPatternTrackingInfo::updateTrackable() {
if (transMat) {
// Keep `trans` as the raw OpenCV pose (rotation unmodified, translation
// scaled). The CV->GL handedness conversion (negate Y and Z rows incl.
// translation) is applied downstream by arglCameraViewRHf when building
// the GL right-handed modelview (matrixGL_RH / getGLViewMatrix).
// Previously, rotation columns 1 and 2 were negated here but the
// translation was not, producing an inconsistent partial conversion
// that left tracked objects behind the camera in OpenGL renderers.
// CV->GL conversion, marker/object side: negate the Y and Z rotation
// COLUMNS (matching ArtoolkitX ARTrackable2d::updateWithTwoDResults).
// Combined with the Y,Z ROW negation applied downstream by arglCameraViewRHf
// (when building matrixGL_RH / getGLViewMatrix), this yields the consistent
// similarity D*R*D (D = diag(1,-1,-1)) -- a proper right-handed marker frame
// with X=right, Y=up, Z=toward the viewer, so AR content placed at +Z pops
// up out of the marker as expected. Without this column negation the frame
// is the raw OpenCV handedness (Z into the marker / away from the camera).
//
// The translation (column 3) is NOT negated here -- it keeps its sign and
// the marker-size scale. arglCameraViewRHf negates the Y,Z translation rows
// downstream, so depth stays in front of the camera (no "behind camera"
// regression); only the rotation handedness is corrected. See WebARKitLib#42.
for (int j = 0; j < 3; j++) {
trans[j][0] = transMat[j][0];
trans[j][1] = transMat[j][1];
trans[j][2] = transMat[j][2];
trans[j][0] = transMat[j][0];
trans[j][1] = -transMat[j][1];
trans[j][2] = -transMat[j][2];
trans[j][3] = (transMat[j][3] * m_scale * 0.001f * 1.64f);
}
}
Expand Down
76 changes: 24 additions & 52 deletions WebARKit/WebARKitTrackers/WebARKitOpticalTracking/WebARKitTracker.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,6 @@ class WebARKitTracker::WebARKitTrackerImpl {
: corners(4), initialized(false), output(17, 0.0), _valid(false), _maxNumberOfMarkersToTrack(1),
_currentlyTrackedMarkers(0), _frameCount(0), _frameSizeX(0),
_frameSizeY(0),
_featureDetectPyrLevel(0),
_featureDetectScaleFactor(cv::Vec2f(1.0f, 1.0f)),
_isDetected(false), _isTracking(false), numMatches(0),
minNumMatches(MIN_NUM_MATCHES), _nn_match_ratio(0.7f), _trackVizActive(false),
_trackViz(TrackerVisualization()) {
Expand All @@ -31,19 +29,10 @@ class WebARKitTracker::WebARKitTrackerImpl {
_frameSizeX = frameWidth;
_frameSizeY = frameHeight;

// Calculate image downsamping factor. 0 = no size change, 1 = half width and height, 2 = quarter width and height etc.
double xmin_log2 = std::log2(static_cast<double>(featureImageMinSize.width));
double ymin_log2 = std::log2(static_cast<double>(featureImageMinSize.height));
_featureDetectPyrLevel = std::min(std::floor(std::log2(static_cast<double>(_frameSizeX)) - xmin_log2), std::floor(std::log2(static_cast<double>(_frameSizeY)) - ymin_log2));

// Calculate the exact scale factor using the same calculation pyrDown uses.
int xScaled = _frameSizeX;
int yScaled = _frameSizeY;
for (int i = 1; i <= _featureDetectPyrLevel; i++) {
xScaled = (xScaled + 1) / 2;
yScaled = (yScaled + 1) / 2;
_featureDetectScaleFactor = cv::Vec2f((float)_frameSizeX / (float)xScaled, (float)_frameSizeY / (float)yScaled);
}
// NOTE: feature detection runs on the FULL-resolution frame (see resetTracking).
// ArtoolkitX's pyramid downsampling + the matching keypoint rescale are
// intentionally not used here; restoring them for performance is tracked in
// webarkit/WebARKitLib#44.

setDetectorType(trackerType);
if (trackerType == webarkit::TEBLID_TRACKER) {
Expand Down Expand Up @@ -340,31 +329,20 @@ class WebARKitTracker::WebARKitTrackerImpl {
cv::Mat frameDescr;
std::vector<cv::KeyPoint> frameKeyPts;

// This if cond. doesn't works as expected in artoolkitx. This make the tracking process unstable.
// if (_currentlyTrackedMarkers < _maxNumberOfMarkersToTrack) {
/*cv::Mat detectionFrame;
if (_featureDetectPyrLevel < 1) {
detectionFrame = frame;
} else {
cv::Mat srcFrame = frame;
for (int pyrLevel = 1; pyrLevel <= _featureDetectPyrLevel; pyrLevel++) {
cv::pyrDown(srcFrame, detectionFrame, cv::Size(0, 0));
srcFrame = detectionFrame;
}
}*/

cv::Mat featureMask = createFeatureMask(frame);

if (!extractFeatures(frame, featureMask, frameKeyPts, frameDescr)) {
WEBARKIT_LOGe("No features detected in extractFeatures!\n");
// return false;
};
//if (!_isDetected) {
WEBARKIT_LOGd("frame KeyPoints size: %d\n", frameKeyPts.size());
if (static_cast<int>(frameKeyPts.size()) > minRequiredDetectedFeatures) {
MatchFeatures(frameKeyPts, frameDescr);
}
//} ref -> if (_currentlyTrackedMarkers < _maxNumberOfMarkersToTrack) {
// Feature detection runs on the FULL-resolution frame, every frame -- no
// pyramid downsampling and no "skip detection while already tracking" guard.
// This favours re-acquisition stability over per-frame cost. Restoring the
// ArtoolkitX downsampling + detection guard for performance is tracked in
// webarkit/WebARKitLib#44.
cv::Mat featureMask = createFeatureMask(frame);

if (!extractFeatures(frame, featureMask, frameKeyPts, frameDescr)) {
WEBARKIT_LOGe("No features detected in extractFeatures!\n");
}
WEBARKIT_LOGd("frame KeyPoints size: %d\n", frameKeyPts.size());
if (static_cast<int>(frameKeyPts.size()) > minRequiredDetectedFeatures) {
MatchFeatures(frameKeyPts, frameDescr);
}
int i = 0;
if (_isDetected) {
WEBARKIT_LOGd("Start tracking!\n");
Expand Down Expand Up @@ -476,11 +454,8 @@ class WebARKitTracker::WebARKitTrackerImpl {
// } // end for cycle

if (maxMatches > 0) {
for (int i = 0; i < finalMatched1.size(); i++) {
finalMatched1[i].pt.x *= _featureDetectScaleFactor[0];
finalMatched1[i].pt.y *= _featureDetectScaleFactor[1];
}

// Matched keypoints are already in full-frame coordinates (full-res
// detection), so no rescale is applied here. See webarkit/WebARKitLib#44.
homography::WebARKitHomographyInfo homoInfo =
getHomographyInliers(Points(finalMatched2), Points(finalMatched1));
if (homoInfo.validHomography) {
Expand Down Expand Up @@ -687,10 +662,11 @@ class WebARKitTracker::WebARKitTrackerImpl {
featureMask = cv::Mat::ones(frame.size(), frame.type());
}
std::vector<std::vector<cv::Point>> contours(1);
// Full-res detection: the warped bbox is already in frame coordinates,
// matching the full-res mask. (No /scaleFactor; see webarkit/WebARKitLib#44.)
for (int j = 0; j < 4; j++) {
// contours[0].push_back(cv::Point(_trackables[i]._bBoxTransformed[j].x/_featureDetectScaleFactor[0],_trackables[i]._bBoxTransformed[j].y/_featureDetectScaleFactor[1]));
contours[0].push_back(cv::Point(_bBoxTransformed[j].x/_featureDetectScaleFactor[0],_bBoxTransformed[j].y/_featureDetectScaleFactor[1]));
}
contours[0].push_back(cv::Point(_bBoxTransformed[j].x, _bBoxTransformed[j].y));
}
drawContours(featureMask, contours, 0, cv::Scalar(0), -1, 8);
}
return featureMask;
Expand All @@ -704,10 +680,6 @@ class WebARKitTracker::WebARKitTrackerImpl {

int _frameSizeX;
int _frameSizeY;
/// Pyramid level used in downsampling incoming image for feature matching. 0 = no size change, 1 = half width/height, 2 = quarter width/heigh etc.
int _featureDetectPyrLevel;
/// Scale factor applied to images used for feature matching. Will be 2^_featureDetectPyrLevel.
cv::Vec2f _featureDetectScaleFactor;

bool _valid;

Expand Down
4 changes: 2 additions & 2 deletions tests/webarkit_test.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,7 @@ TEST(WebARKitGLTest, TestCameraProjectionMatrix) {
std::array<double, 9> camera_mat = camera.getCameraData();
std::array<double, 16> projectionMatrix = {0.0};
webarkit::cameraProjectionMatrix(camera_mat, 0.01, 100.0, width, height, projectionMatrix);
EXPECT_EQ(projectionMatrix[0], -1.7851850084276433);
EXPECT_EQ(projectionMatrix[0], 1.7851850084276433);
EXPECT_EQ(projectionMatrix[5], 2.3802466779035241);
EXPECT_EQ(projectionMatrix[10], -1.0002000200020003);
EXPECT_EQ(projectionMatrix[11], -1.0);
Expand Down Expand Up @@ -147,7 +147,7 @@ TEST(WebARKitTest, CheckCameraProjectionMatrix) {
manager.initialiseBase(webarkit::TRACKER_TYPE::AKAZE_TRACKER, 640, 480);
// Check if the cameraProjectionMatrix is correct
std::array<double, 16> camProjectionMatrix = manager.getCameraProjectionMatrix();
EXPECT_EQ(camProjectionMatrix[0], -1.7851850084276433);
EXPECT_EQ(camProjectionMatrix[0], 1.7851850084276433);
EXPECT_EQ(camProjectionMatrix[5], 2.3802466779035241);
EXPECT_EQ(camProjectionMatrix[10], -1.0002000200020003);
EXPECT_EQ(camProjectionMatrix[11], -1.0);
Expand Down