feat(#44): detect features on downsampled frame (part A) - #52
Merged
Conversation
Restore ArtoolkitX-style detection-side pyramid downsampling: feature detection now runs on a pyrDown'd copy of the live frame (the "detectionFrame"), and matched frame keypoints are scaled back to full-frame coordinates before the homography fit. - initialize(): compute _featureDetectPyrLevel from featureImageMinSize (clamped >= 0) and the exact cv::pyrDown scale factor via the iterative (x+1)/2 rounding, so the keypoint rescale is sub-pixel accurate. - processFrame(): build detectionFrame via _featureDetectPyrLevel pyrDown steps; run createFeatureMask/extractFeatures on it. - MatchFeatures(): rescale matched frame keypoints by _featureDetectScaleFactor before getHomographyInliers. - createFeatureMask(): divide the warped bbox by the scale factor so the tracked-marker exclusion lands in downsampled coordinates. Level 0 (frame <= featureImageMinSize, e.g. 640x480) => detectionFrame == frame and factor 1.0, so that path is byte-identical to full-res detection. The reference image stays detected at full resolution (initTracker), only the live frame is downsampled, matching ArtoolkitX OCVT. This is part A of webarkit#44. The "skip detection while tracking" guard is a separate follow-up (part B). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
kalwalt
added a commit
to webarkit/webarkit-testing
that referenced
this pull request
Jun 18, 2026
…xample Bump WebARKitLib to the part-A downsampling change (webarkit/WebARKitLib#52) and rebuild build/ + dist/. Feature detection now runs on a pyrDown'd copy of the live frame; matched keypoints are rescaled to full-frame coords. - examples: point the Teblid static-image example at a marker-fills-frame 1920x1440 demo image (pinball-demo-big.jpg) so HD downsampling is exercised (pyrLevel 1, scale 2.0) and the marker still acquires. The old pinball-demo.jpg (small marker in a large frame) falls below the detector threshold once downsampled. - docs: add docs/design-detection-downsampling.md (understanding, ArtoolkitX reference, decision log, the four edits, testing, risks, non-goals). Verified: Marker tracked, 1000 matches, content centered (#38) and right-handed (#42/#46), no #43-style doubling. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
kalwalt
added a commit
that referenced
this pull request
Jun 19, 2026
Guard the per-frame feature detection (extractFeatures + descriptor matching) with `if (!_isTracking)`. Once optical flow holds a lock, detection is skipped and the pose is maintained by optical flow + template matching; when the marker is lost (_isTracking cleared in runOpticalFlow / RunTemplateMatching), detection resumes on the next frame to re-acquire. Detection is the dominant per-frame cost, so skipping it in the steady tracking state is a large win: at 640x480 the webcam example goes from ~10-15 fps (detecting every frame) to ~45 fps while tracking. This is part B of #44; part A (detection-side pyramid downsampling) landed in #52. The guard is on _isTracking rather than ArtoolkitX's `_currentlyTrackedMarkers < _maxNumberOfMarkersToTrack`: _isDetected is reset to false every frame, and optical flow is skipped on the first detection frame (_frameCount == 0), so a counter-based guard would skip both detection and optical flow on the following frame and freeze (this bites the static-image example, which feeds the same frame repeatedly). Gating on _isTracking re-detects until optical flow actually holds a lock. The optical-flow/template/pose paths (gated on _isDetected || _isTracking) and #46 tracking-loss are unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
kalwalt
added a commit
to webarkit/webarkit-testing
that referenced
this pull request
Jun 21, 2026
…xample Bump WebARKitLib to the part-A downsampling change (webarkit/WebARKitLib#52) and rebuild build/ + dist/. Feature detection now runs on a pyrDown'd copy of the live frame; matched keypoints are rescaled to full-frame coords. - examples: point the Teblid static-image example at a marker-fills-frame 1920x1440 demo image (pinball-demo-big.jpg) so HD downsampling is exercised (pyrLevel 1, scale 2.0) and the marker still acquires. The old pinball-demo.jpg (small marker in a large frame) falls below the detector threshold once downsampled. - docs: add docs/design-detection-downsampling.md (understanding, ArtoolkitX reference, decision log, the four edits, testing, risks, non-goals). Verified: Marker tracked, 1000 matches, content centered (#38) and right-handed (#42/#46), no #43-style doubling. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Restore ArtoolkitX-style detection-side pyramid downsampling (part A of #44). Feature detection now runs on a
pyrDown'd copy of the live frame (detectionFrame); matched frame keypoints are scaled back to full-frame coordinates before the homography fit.Why
Detection previously ran at full resolution every frame — expensive on HD inputs. Downsampling cuts that cost on frames larger than
featureImageMinSize(640×480). This also makes the keypoint rescale from #43 correct by construction (detection is genuinely downsampled), instead of the_featureDetectScaleFactor = 1workaround.Changes (
WebARKitTracker.cpp)initialize()— compute_featureDetectPyrLevelfromfeatureImageMinSize(clamped ≥ 0) and the exactcv::pyrDownscale factor via the iterative(x+1)/2rounding (sub-pixel-accurate rescale).processFrame()— builddetectionFramevia_featureDetectPyrLevelpyrDownsteps; runcreateFeatureMask/extractFeatureson it.MatchFeatures()— rescale matched frame keypoints by_featureDetectScaleFactorbeforegetHomographyInliers.createFeatureMask()— divide the warped bbox by the scale factor so the tracked-marker exclusion lands in downsampled coordinates.Compatibility
featureImageMinSize, e.g. 640×480) ⇒detectionFrame == frame, factor1.0⇒ byte-identical to full-res detection.initTracker); only the live frame is downsampled — matches ArtoolkitX OCVT.Testing
Verified on the Teblid static-image example at 1920×1440 (pyrLevel 1, scale factor 2.0):
Marker tracked! Num. matches: 1000, content correctly centered (#38) and right-handed (#42/#46), no #43-style doubling.Scope
Part A only. The "skip detection while tracking" guard is a separate follow-up (part B).
🤖 Generated with Claude Code