feat(computer-use): OCR pictureOnly fallback (macOS Vision) - #256

Merged
Tryanks merged 1 commit into
mainfrom
feat/123-ocr-fallback
Aug 22, 2026
Merged

feat(computer-use): OCR pictureOnly fallback (macOS Vision)#256
Tryanks merged 1 commit into
mainfrom
feat/123-ocr-fallback

Conversation

@Tryanks

@TryanksTryanks commented Aug 22, 2026

Copy link
Copy Markdown
Owner

Implements the OCR track of #123 (does not close it — the Windows/UIAutomation backend remains).

  • Trigger: frame ≥ 20,000 sq screen points with < 3 text-bearing AX descendants (root title excluded). Adequate trees never invoke Vision.
  • Synthesis: accurate-mode VNRecognizeTextRequest lines become static_text leaves with picture_only: true, [click,pictureOnly] outline rendering, absolute screen frames (normalized lower-left Vision boxes mapped, finite-checked, clipped), ordered top-to-bottom/left-to-right for stable refs. Distinct ref/diff identity from AX nodes; search/inspect/read_text work on them; clicks go through the existing CGEvent path at box center; other actions on OCR refs are rejected.
  • Perf/gating: one capture per observation, shared between OCR and the returned screenshot; imageModeauto/always/never gates OCR exactly like screenshots; OCR failure degrades to the AX tree + screenshot.
  • Deps: macOS-only objc2-vision 0.3.2 (minimal features) + matching objc2/foundation features.
  • Tests: 15 platform-neutral tests (trigger rule, synthesis/filtering, ordering/provenance, coordinate mapping, click prep, action rejection). Live end-to-end was blocked by TCC in the automated environment (no Screen Recording grant) — honestly reported; the unit layer covers the pure logic and the capture path is the existing screenshot path.
  • docs/computer-use.md deviations updated: OCR fallback shipped; Windows backend + CDP roots remain.

Gates: fmt / clippy -D warnings / full workspace tests — green locally.

Part of #123

🤖 Generated with Claude Code

@Tryanks@claude
feat(computer-use): OCR pictureOnly fallback for text-sparse AX trees…
50a0d25
… (macOS)
When an observed frame ≥20k sq pt has <3 text-bearing AX descendants,
the window is captured once (shared with the returned screenshot) and
VNRecognizeTextRequest synthesizes clickable static_text leaves with
picture_only provenance, absolute screen frames, and stable
top-to-bottom refs. Adequate trees never invoke Vision; imageMode
auto/always/never gates OCR consistently with screenshots. Unlocks
Electron shells, canvas UIs, and games where the AX-first approach
returned nothing.
Part of #123 (Windows/UIAutomation track remains)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Tryanks
Tryanks enabled auto-merge (squash) August 22, 2026 16:26
@TryanksTryanks mentioned this pull request Aug 22, 2026
Closed
@Tryanks
Tryanks merged commit d84b928 into mainAug 22, 2026
3 checks passed
@Tryanks
Tryanks deleted the feat/123-ocr-fallback branch August 22, 2026 16:33
@TryanksTryanks mentioned this pull request Aug 22, 2026
Merged
Tryanks added a commit that referenced this pull request Aug 22, 2026
@Tryanks
Revert "feat(computer-use): OCR pictureOnly fallback for text-sparse …
b361d80
…AX trees (macOS) (#256)" (#257)
This reverts commit d84b928.
@TryanksTryanks mentioned this pull request Aug 23, 2026
Closed
12 tasks
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@Tryanks
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
feat(computer-use): OCR pictureOnly fallback (macOS Vision) - #256
Merged
Tryanks merged 1 commit into
mainfrom 
feat/123-ocr-fallback
Aug 22, 2026
Merged
feat(computer-use): OCR pictureOnly fallback (macOS Vision)#256
Tryanks merged 1 commit into
mainfrom 
feat/123-ocr-fallback
Conversation
@Tryanks
@TryanksTryanks
commented
Aug 22, 2026

Copy link


Copy Markdown

Owner
Implements the OCR track of #123 (does not close it — the Windows/UIAutomation backend remains).
  • Trigger: frame ≥ 20,000 sq screen points with < 3 text-bearing AX descendants (root title excluded). Adequate trees never invoke Vision.
  • Synthesis: accurate-mode VNRecognizeTextRequest lines become static_text leaves with picture_only: true, [click,pictureOnly] outline rendering, absolute screen frames (normalized lower-left Vision boxes mapped, finite-checked, clipped), ordered top-to-bottom/left-to-right for stable refs. Distinct ref/diff identity from AX nodes; search/inspect/read_text work on them; clicks go through the existing CGEvent path at box center; other actions on OCR refs are rejected.
  • Perf/gating: one capture per observation, shared between OCR and the returned screenshot; imageModeauto/always/never gates OCR exactly like screenshots; OCR failure degrades to the AX tree + screenshot.
  • Deps: macOS-only objc2-vision 0.3.2 (minimal features) + matching objc2/foundation features.
  • Tests: 15 platform-neutral tests (trigger rule, synthesis/filtering, ordering/provenance, coordinate mapping, click prep, action rejection). Live end-to-end was blocked by TCC in the automated environment (no Screen Recording grant) — honestly reported; the unit layer covers the pure logic and the capture path is the existing screenshot path.
  • docs/computer-use.md deviations updated: OCR fallback shipped; Windows backend + CDP roots remain.
Gates: fmt / clippy -D warnings / full workspace tests — green locally.
Part of #123
🤖 Generated with Claude Code
@Tryanks@claude
feat(computer-use): OCR pictureOnly fallback for text-sparse AX trees…
50a0d25
… (macOS)
When an observed frame ≥20k sq pt has <3 text-bearing AX descendants,
the window is captured once (shared with the returned screenshot) and
VNRecognizeTextRequest synthesizes clickable static_text leaves with
picture_only provenance, absolute screen frames, and stable
top-to-bottom refs. Adequate trees never invoke Vision; imageMode
auto/always/never gates OCR consistently with screenshots. Unlocks
Electron shells, canvas UIs, and games where the AX-first approach
returned nothing.
Part of #123 (Windows/UIAutomation track remains)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Tryanks
Tryanks
enabled auto-merge (squash)
August 22, 2026 16:26
@TryanksTryanks
mentioned this pull request
Aug 22, 2026

Closed

@Tryanks
Tryanks
merged commit d84b928
into
mainAug 22, 2026

3 checks passed

@Tryanks
Tryanks
deleted the

feat/123-ocr-fallback

branch
August 22, 2026 16:33
@TryanksTryanks
mentioned this pull request
Aug 22, 2026
 Merged

Tryanks added a commit
that referenced
this pull request
Aug 22, 2026
@Tryanks
Revert "feat(computer-use): OCR pictureOnly fallback for text-sparse …
b361d80
…AX trees (macOS) (#256)" (#257)
This reverts commit d84b928.
@TryanksTryanks
mentioned this pull request
Aug 23, 2026

Closed

12 tasks
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


No reviews


Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

1 participant

@Tryanks

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
feat(computer-use): OCR pictureOnly fallback (macOS Vision) - #256
Merged
Tryanks merged 1 commit into
mainfrom 
feat/123-ocr-fallback
Aug 22, 2026
Merged
feat(computer-use): OCR pictureOnly fallback (macOS Vision)#256
Tryanks merged 1 commit into
mainfrom 
feat/123-ocr-fallback
Conversation
@Tryanks
@TryanksTryanks
commented
Aug 22, 2026

Copy link


Copy Markdown

Owner
Implements the OCR track of #123 (does not close it — the Windows/UIAutomation backend remains).
  • Trigger: frame ≥ 20,000 sq screen points with < 3 text-bearing AX descendants (root title excluded). Adequate trees never invoke Vision.
  • Synthesis: accurate-mode VNRecognizeTextRequest lines become static_text leaves with picture_only: true, [click,pictureOnly] outline rendering, absolute screen frames (normalized lower-left Vision boxes mapped, finite-checked, clipped), ordered top-to-bottom/left-to-right for stable refs. Distinct ref/diff identity from AX nodes; search/inspect/read_text work on them; clicks go through the existing CGEvent path at box center; other actions on OCR refs are rejected.
  • Perf/gating: one capture per observation, shared between OCR and the returned screenshot; imageModeauto/always/never gates OCR exactly like screenshots; OCR failure degrades to the AX tree + screenshot.
  • Deps: macOS-only objc2-vision 0.3.2 (minimal features) + matching objc2/foundation features.
  • Tests: 15 platform-neutral tests (trigger rule, synthesis/filtering, ordering/provenance, coordinate mapping, click prep, action rejection). Live end-to-end was blocked by TCC in the automated environment (no Screen Recording grant) — honestly reported; the unit layer covers the pure logic and the capture path is the existing screenshot path.
  • docs/computer-use.md deviations updated: OCR fallback shipped; Windows backend + CDP roots remain.
Gates: fmt / clippy -D warnings / full workspace tests — green locally.
Part of #123
🤖 Generated with Claude Code
@Tryanks@claude
feat(computer-use): OCR pictureOnly fallback for text-sparse AX trees…
50a0d25
… (macOS)
When an observed frame ≥20k sq pt has <3 text-bearing AX descendants,
the window is captured once (shared with the returned screenshot) and
VNRecognizeTextRequest synthesizes clickable static_text leaves with
picture_only provenance, absolute screen frames, and stable
top-to-bottom refs. Adequate trees never invoke Vision; imageMode
auto/always/never gates OCR consistently with screenshots. Unlocks
Electron shells, canvas UIs, and games where the AX-first approach
returned nothing.
Part of #123 (Windows/UIAutomation track remains)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Tryanks
Tryanks
enabled auto-merge (squash)
August 22, 2026 16:26
@TryanksTryanks
mentioned this pull request
Aug 22, 2026

Closed

@Tryanks
Tryanks
merged commit d84b928
into
mainAug 22, 2026

3 checks passed

@Tryanks
Tryanks
deleted the

feat/123-ocr-fallback

branch
August 22, 2026 16:33
@TryanksTryanks
mentioned this pull request
Aug 22, 2026
 Merged

Tryanks added a commit
that referenced
this pull request
Aug 22, 2026
@Tryanks
Revert "feat(computer-use): OCR pictureOnly fallback for text-sparse …
b361d80
…AX trees (macOS) (#256)" (#257)
This reverts commit d84b928.
@TryanksTryanks
mentioned this pull request
Aug 23, 2026

Closed

12 tasks
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


No reviews


Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

1 participant

@Tryanks

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
feat(computer-use): OCR pictureOnly fallback (macOS Vision) - #256
Merged
Tryanks merged 1 commit into
mainfrom 
feat/123-ocr-fallback
Aug 22, 2026
Merged
feat(computer-use): OCR pictureOnly fallback (macOS Vision)#256
Tryanks merged 1 commit into
mainfrom 
feat/123-ocr-fallback
Conversation
@Tryanks
@TryanksTryanks
commented
Aug 22, 2026

Copy link


Copy Markdown

Owner
Implements the OCR track of #123 (does not close it — the Windows/UIAutomation backend remains).
  • Trigger: frame ≥ 20,000 sq screen points with < 3 text-bearing AX descendants (root title excluded). Adequate trees never invoke Vision.
  • Synthesis: accurate-mode VNRecognizeTextRequest lines become static_text leaves with picture_only: true, [click,pictureOnly] outline rendering, absolute screen frames (normalized lower-left Vision boxes mapped, finite-checked, clipped), ordered top-to-bottom/left-to-right for stable refs. Distinct ref/diff identity from AX nodes; search/inspect/read_text work on them; clicks go through the existing CGEvent path at box center; other actions on OCR refs are rejected.
  • Perf/gating: one capture per observation, shared between OCR and the returned screenshot; imageModeauto/always/never gates OCR exactly like screenshots; OCR failure degrades to the AX tree + screenshot.
  • Deps: macOS-only objc2-vision 0.3.2 (minimal features) + matching objc2/foundation features.
  • Tests: 15 platform-neutral tests (trigger rule, synthesis/filtering, ordering/provenance, coordinate mapping, click prep, action rejection). Live end-to-end was blocked by TCC in the automated environment (no Screen Recording grant) — honestly reported; the unit layer covers the pure logic and the capture path is the existing screenshot path.
  • docs/computer-use.md deviations updated: OCR fallback shipped; Windows backend + CDP roots remain.
Gates: fmt / clippy -D warnings / full workspace tests — green locally.
Part of #123
🤖 Generated with Claude Code
@Tryanks@claude
feat(computer-use): OCR pictureOnly fallback for text-sparse AX trees…
50a0d25
… (macOS)
When an observed frame ≥20k sq pt has <3 text-bearing AX descendants,
the window is captured once (shared with the returned screenshot) and
VNRecognizeTextRequest synthesizes clickable static_text leaves with
picture_only provenance, absolute screen frames, and stable
top-to-bottom refs. Adequate trees never invoke Vision; imageMode
auto/always/never gates OCR consistently with screenshots. Unlocks
Electron shells, canvas UIs, and games where the AX-first approach
returned nothing.
Part of #123 (Windows/UIAutomation track remains)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Tryanks
Tryanks
enabled auto-merge (squash)
August 22, 2026 16:26
@TryanksTryanks
mentioned this pull request
Aug 22, 2026

Closed

@Tryanks
Tryanks
merged commit d84b928
into
mainAug 22, 2026

3 checks passed

@Tryanks
Tryanks
deleted the

feat/123-ocr-fallback

branch
August 22, 2026 16:33
@TryanksTryanks
mentioned this pull request
Aug 22, 2026
 Merged

Tryanks added a commit
that referenced
this pull request
Aug 22, 2026
@Tryanks
Revert "feat(computer-use): OCR pictureOnly fallback for text-sparse …
b361d80
…AX trees (macOS) (#256)" (#257)
This reverts commit d84b928.
@TryanksTryanks
mentioned this pull request
Aug 23, 2026

Closed

12 tasks
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


No reviews


Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

1 participant

@Tryanks

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
feat(computer-use): OCR pictureOnly fallback (macOS Vision) - #256
Merged
Tryanks merged 1 commit into
mainfrom 
feat/123-ocr-fallback
Aug 22, 2026
Merged
feat(computer-use): OCR pictureOnly fallback (macOS Vision)#256
Tryanks merged 1 commit into
mainfrom 
feat/123-ocr-fallback
Conversation
@Tryanks
@TryanksTryanks
commented
Aug 22, 2026

Copy link


Copy Markdown

Owner
Implements the OCR track of #123 (does not close it — the Windows/UIAutomation backend remains).
  • Trigger: frame ≥ 20,000 sq screen points with < 3 text-bearing AX descendants (root title excluded). Adequate trees never invoke Vision.
  • Synthesis: accurate-mode VNRecognizeTextRequest lines become static_text leaves with picture_only: true, [click,pictureOnly] outline rendering, absolute screen frames (normalized lower-left Vision boxes mapped, finite-checked, clipped), ordered top-to-bottom/left-to-right for stable refs. Distinct ref/diff identity from AX nodes; search/inspect/read_text work on them; clicks go through the existing CGEvent path at box center; other actions on OCR refs are rejected.
  • Perf/gating: one capture per observation, shared between OCR and the returned screenshot; imageModeauto/always/never gates OCR exactly like screenshots; OCR failure degrades to the AX tree + screenshot.
  • Deps: macOS-only objc2-vision 0.3.2 (minimal features) + matching objc2/foundation features.
  • Tests: 15 platform-neutral tests (trigger rule, synthesis/filtering, ordering/provenance, coordinate mapping, click prep, action rejection). Live end-to-end was blocked by TCC in the automated environment (no Screen Recording grant) — honestly reported; the unit layer covers the pure logic and the capture path is the existing screenshot path.
  • docs/computer-use.md deviations updated: OCR fallback shipped; Windows backend + CDP roots remain.
Gates: fmt / clippy -D warnings / full workspace tests — green locally.
Part of #123
🤖 Generated with Claude Code
@Tryanks@claude
feat(computer-use): OCR pictureOnly fallback for text-sparse AX trees…
50a0d25
… (macOS)
When an observed frame ≥20k sq pt has <3 text-bearing AX descendants,
the window is captured once (shared with the returned screenshot) and
VNRecognizeTextRequest synthesizes clickable static_text leaves with
picture_only provenance, absolute screen frames, and stable
top-to-bottom refs. Adequate trees never invoke Vision; imageMode
auto/always/never gates OCR consistently with screenshots. Unlocks
Electron shells, canvas UIs, and games where the AX-first approach
returned nothing.
Part of #123 (Windows/UIAutomation track remains)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Tryanks
Tryanks
enabled auto-merge (squash)
August 22, 2026 16:26
@TryanksTryanks
mentioned this pull request
Aug 22, 2026

Closed

@Tryanks
Tryanks
merged commit d84b928
into
mainAug 22, 2026

3 checks passed

@Tryanks
Tryanks
deleted the

feat/123-ocr-fallback

branch
August 22, 2026 16:33
@TryanksTryanks
mentioned this pull request
Aug 22, 2026
 Merged

Tryanks added a commit
that referenced
this pull request
Aug 22, 2026
@Tryanks
Revert "feat(computer-use): OCR pictureOnly fallback for text-sparse …
b361d80
…AX trees (macOS) (#256)" (#257)
This reverts commit d84b928.
@TryanksTryanks
mentioned this pull request
Aug 23, 2026

Closed

12 tasks
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


No reviews


Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

1 participant

@Tryanks

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
feat(computer-use): OCR pictureOnly fallback (macOS Vision) - #256
Merged
Tryanks merged 1 commit into
mainfrom 
feat/123-ocr-fallback
Aug 22, 2026
Merged
feat(computer-use): OCR pictureOnly fallback (macOS Vision)#256
Tryanks merged 1 commit into
mainfrom 
feat/123-ocr-fallback
Conversation
@Tryanks
@TryanksTryanks
commented
Aug 22, 2026

Copy link


Copy Markdown

Owner
Implements the OCR track of #123 (does not close it — the Windows/UIAutomation backend remains).
  • Trigger: frame ≥ 20,000 sq screen points with < 3 text-bearing AX descendants (root title excluded). Adequate trees never invoke Vision.
  • Synthesis: accurate-mode VNRecognizeTextRequest lines become static_text leaves with picture_only: true, [click,pictureOnly] outline rendering, absolute screen frames (normalized lower-left Vision boxes mapped, finite-checked, clipped), ordered top-to-bottom/left-to-right for stable refs. Distinct ref/diff identity from AX nodes; search/inspect/read_text work on them; clicks go through the existing CGEvent path at box center; other actions on OCR refs are rejected.
  • Perf/gating: one capture per observation, shared between OCR and the returned screenshot; imageModeauto/always/never gates OCR exactly like screenshots; OCR failure degrades to the AX tree + screenshot.
  • Deps: macOS-only objc2-vision 0.3.2 (minimal features) + matching objc2/foundation features.
  • Tests: 15 platform-neutral tests (trigger rule, synthesis/filtering, ordering/provenance, coordinate mapping, click prep, action rejection). Live end-to-end was blocked by TCC in the automated environment (no Screen Recording grant) — honestly reported; the unit layer covers the pure logic and the capture path is the existing screenshot path.
  • docs/computer-use.md deviations updated: OCR fallback shipped; Windows backend + CDP roots remain.
Gates: fmt / clippy -D warnings / full workspace tests — green locally.
Part of #123
🤖 Generated with Claude Code
@Tryanks@claude
feat(computer-use): OCR pictureOnly fallback for text-sparse AX trees…
50a0d25
… (macOS)
When an observed frame ≥20k sq pt has <3 text-bearing AX descendants,
the window is captured once (shared with the returned screenshot) and
VNRecognizeTextRequest synthesizes clickable static_text leaves with
picture_only provenance, absolute screen frames, and stable
top-to-bottom refs. Adequate trees never invoke Vision; imageMode
auto/always/never gates OCR consistently with screenshots. Unlocks
Electron shells, canvas UIs, and games where the AX-first approach
returned nothing.
Part of #123 (Windows/UIAutomation track remains)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Tryanks
Tryanks
enabled auto-merge (squash)
August 22, 2026 16:26
@TryanksTryanks
mentioned this pull request
Aug 22, 2026

Closed

@Tryanks
Tryanks
merged commit d84b928
into
mainAug 22, 2026

3 checks passed

@Tryanks
Tryanks
deleted the

feat/123-ocr-fallback

branch
August 22, 2026 16:33
@TryanksTryanks
mentioned this pull request
Aug 22, 2026
 Merged

Tryanks added a commit
that referenced
this pull request
Aug 22, 2026
@Tryanks
Revert "feat(computer-use): OCR pictureOnly fallback for text-sparse …
b361d80
…AX trees (macOS) (#256)" (#257)
This reverts commit d84b928.
@TryanksTryanks
mentioned this pull request
Aug 23, 2026

Closed

12 tasks
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


No reviews


Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

1 participant

@Tryanks

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
feat(computer-use): OCR pictureOnly fallback (macOS Vision) - #256
Merged
Tryanks merged 1 commit into
mainfrom 
feat/123-ocr-fallback
Aug 22, 2026
Merged
feat(computer-use): OCR pictureOnly fallback (macOS Vision)#256
Tryanks merged 1 commit into
mainfrom 
feat/123-ocr-fallback
Conversation
@Tryanks
@TryanksTryanks
commented
Aug 22, 2026

Copy link


Copy Markdown

Owner
Implements the OCR track of #123 (does not close it — the Windows/UIAutomation backend remains).
  • Trigger: frame ≥ 20,000 sq screen points with < 3 text-bearing AX descendants (root title excluded). Adequate trees never invoke Vision.
  • Synthesis: accurate-mode VNRecognizeTextRequest lines become static_text leaves with picture_only: true, [click,pictureOnly] outline rendering, absolute screen frames (normalized lower-left Vision boxes mapped, finite-checked, clipped), ordered top-to-bottom/left-to-right for stable refs. Distinct ref/diff identity from AX nodes; search/inspect/read_text work on them; clicks go through the existing CGEvent path at box center; other actions on OCR refs are rejected.
  • Perf/gating: one capture per observation, shared between OCR and the returned screenshot; imageModeauto/always/never gates OCR exactly like screenshots; OCR failure degrades to the AX tree + screenshot.
  • Deps: macOS-only objc2-vision 0.3.2 (minimal features) + matching objc2/foundation features.
  • Tests: 15 platform-neutral tests (trigger rule, synthesis/filtering, ordering/provenance, coordinate mapping, click prep, action rejection). Live end-to-end was blocked by TCC in the automated environment (no Screen Recording grant) — honestly reported; the unit layer covers the pure logic and the capture path is the existing screenshot path.
  • docs/computer-use.md deviations updated: OCR fallback shipped; Windows backend + CDP roots remain.
Gates: fmt / clippy -D warnings / full workspace tests — green locally.
Part of #123
🤖 Generated with Claude Code
@Tryanks@claude
feat(computer-use): OCR pictureOnly fallback for text-sparse AX trees…
50a0d25
… (macOS)
When an observed frame ≥20k sq pt has <3 text-bearing AX descendants,
the window is captured once (shared with the returned screenshot) and
VNRecognizeTextRequest synthesizes clickable static_text leaves with
picture_only provenance, absolute screen frames, and stable
top-to-bottom refs. Adequate trees never invoke Vision; imageMode
auto/always/never gates OCR consistently with screenshots. Unlocks
Electron shells, canvas UIs, and games where the AX-first approach
returned nothing.
Part of #123 (Windows/UIAutomation track remains)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Tryanks
Tryanks
enabled auto-merge (squash)
August 22, 2026 16:26
@TryanksTryanks
mentioned this pull request
Aug 22, 2026

Closed

@Tryanks
Tryanks
merged commit d84b928
into
mainAug 22, 2026

3 checks passed

@Tryanks
Tryanks
deleted the

feat/123-ocr-fallback

branch
August 22, 2026 16:33
@TryanksTryanks
mentioned this pull request
Aug 22, 2026
 Merged

Tryanks added a commit
that referenced
this pull request
Aug 22, 2026
@Tryanks
Revert "feat(computer-use): OCR pictureOnly fallback for text-sparse …
b361d80
…AX trees (macOS) (#256)" (#257)
This reverts commit d84b928.
@TryanksTryanks
mentioned this pull request
Aug 23, 2026

Closed

12 tasks
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


No reviews


Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

1 participant

@Tryanks

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
feat(computer-use): OCR pictureOnly fallback (macOS Vision) - #256
Merged
Tryanks merged 1 commit into
mainfrom 
feat/123-ocr-fallback
Aug 22, 2026
Merged
feat(computer-use): OCR pictureOnly fallback (macOS Vision)#256
Tryanks merged 1 commit into
mainfrom 
feat/123-ocr-fallback
Conversation
@Tryanks
@TryanksTryanks
commented
Aug 22, 2026

Copy link


Copy Markdown

Owner
Implements the OCR track of #123 (does not close it — the Windows/UIAutomation backend remains).
  • Trigger: frame ≥ 20,000 sq screen points with < 3 text-bearing AX descendants (root title excluded). Adequate trees never invoke Vision.
  • Synthesis: accurate-mode VNRecognizeTextRequest lines become static_text leaves with picture_only: true, [click,pictureOnly] outline rendering, absolute screen frames (normalized lower-left Vision boxes mapped, finite-checked, clipped), ordered top-to-bottom/left-to-right for stable refs. Distinct ref/diff identity from AX nodes; search/inspect/read_text work on them; clicks go through the existing CGEvent path at box center; other actions on OCR refs are rejected.
  • Perf/gating: one capture per observation, shared between OCR and the returned screenshot; imageModeauto/always/never gates OCR exactly like screenshots; OCR failure degrades to the AX tree + screenshot.
  • Deps: macOS-only objc2-vision 0.3.2 (minimal features) + matching objc2/foundation features.
  • Tests: 15 platform-neutral tests (trigger rule, synthesis/filtering, ordering/provenance, coordinate mapping, click prep, action rejection). Live end-to-end was blocked by TCC in the automated environment (no Screen Recording grant) — honestly reported; the unit layer covers the pure logic and the capture path is the existing screenshot path.
  • docs/computer-use.md deviations updated: OCR fallback shipped; Windows backend + CDP roots remain.
Gates: fmt / clippy -D warnings / full workspace tests — green locally.
Part of #123
🤖 Generated with Claude Code
@Tryanks@claude
feat(computer-use): OCR pictureOnly fallback for text-sparse AX trees…
50a0d25
… (macOS)
When an observed frame ≥20k sq pt has <3 text-bearing AX descendants,
the window is captured once (shared with the returned screenshot) and
VNRecognizeTextRequest synthesizes clickable static_text leaves with
picture_only provenance, absolute screen frames, and stable
top-to-bottom refs. Adequate trees never invoke Vision; imageMode
auto/always/never gates OCR consistently with screenshots. Unlocks
Electron shells, canvas UIs, and games where the AX-first approach
returned nothing.
Part of #123 (Windows/UIAutomation track remains)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Tryanks
Tryanks
enabled auto-merge (squash)
August 22, 2026 16:26
@TryanksTryanks
mentioned this pull request
Aug 22, 2026

Closed

@Tryanks
Tryanks
merged commit d84b928
into
mainAug 22, 2026

3 checks passed

@Tryanks
Tryanks
deleted the

feat/123-ocr-fallback

branch
August 22, 2026 16:33
@TryanksTryanks
mentioned this pull request
Aug 22, 2026
 Merged

Tryanks added a commit
that referenced
this pull request
Aug 22, 2026
@Tryanks
Revert "feat(computer-use): OCR pictureOnly fallback for text-sparse …
b361d80
…AX trees (macOS) (#256)" (#257)
This reverts commit d84b928.
@TryanksTryanks
mentioned this pull request
Aug 23, 2026

Closed

12 tasks
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


No reviews


Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

1 participant

@Tryanks