camera: rotate session to portrait so YOLO sees an upright scene - #11
Merged
Conversation
iOS's camera sensor captures in landscape-right by default. With the phone held in portrait, both AVCaptureVideoPreviewLayer and the new AVCaptureVideoDataOutput delivered sideways frames — invisible to the user when only the preview was used, but devastating once we started feeding pixels to an ML model trained on upright COCO images. A jar held vertically in the UI arrived at YOLO as a horizontal bar and got classified as "laptop" or "cell phone" at low confidence. Pin both the preview and the frame stream to 90° (videoRotationAngle on iOS 17+, videoOrientation = .portrait on older builds). With this in place, the same jar lands as "cup 96%" — high enough that the demo no longer needs the tuned-down confidence threshold to surface anything. What you see on the preview is what the model sees, and detection boxes now align with their objects.
pshoukry
pushed a commit
to pshoukry/mob
that referenced
this pull request
Sep 9, 2026
…ixed (mob_new + mob_dev companion commits) Four small lost-in-the-shuffle items closed in this batch. All four were held up by Phase 2 work touching the same files (GenericJam#1/GenericJam#2/GenericJam#4 in live_view_patcher.ex; GenericJam#5 in native_build.ex). GenericJam#1 — Phoenix LiveReload mac_listener warnings: code_reloader/watchers/ live_reload disabled in on-device endpoint config. GenericJam#2 — esbuild/tailwind version-not-configured warnings: versions set via Application.put_env in mob_app.ex before ensure_all_started. GenericJam#4 — port 4200 collisions across multiple Mob LV apps: per-app hash into 4200..4999 via :erlang.phash2(:<app>, 800). GenericJam#5 — deploy auto-pick of iPhone over sim was silent: prints the --device <short-id> alternative when both are connected. GenericJam#3 (WS→longpoll fallback in WKWebView) is investigation, not a fix — deferred. GenericJam#6-GenericJam#11 are larger work (OTP rebuild, AX modifiers, Compose semantics walker, Android 17 SELinux patch). GenericJam#12, GenericJam#13 already fixed earlier. GenericJam#14 is moderate — sim node naming reconciliation between mob_dev's connect.ex and mob_beam.m, deferred.
pshoukry
pushed a commit
to pshoukry/mob
that referenced
this pull request
Sep 9, 2026
Closes mob/issues.md GenericJam#7 + GenericJam#8 — both were AX-conformance bugs in the SwiftUI components that backed `<Slider>` and `<Toggle>` in the renderer. GenericJam#7: `MobSlider` had no `.accessibilityAdjustableAction`. SwiftUI ignores VoiceOver / `accessibilityIncrement` calls without it; the NIF returned :ok but the value never moved. Added the modifier with default step `(max - min) / 10` (matches VoiceOver's default for native UISlider) and a wired callback so on-change events fire too. GenericJam#8: `MobToggle` used SwiftUI's `Toggle("Label", isOn:)` form, which does NOT propagate the label string into the underlying control's `accessibilityLabel`. The AX tree exposed the toggle as a button with empty label, so `Mob.Test.toggle(node, "Notifications")` returned :label_not_found. Added `.accessibilityLabel(label)` after the toggle so the visible text reaches the AX tree. Both fixes are in `ios/MobRootView.swift`. Android counterparts (`MobSlider` / `MobToggle` in MobBridgeKt) need analogous `Modifier.semantics { setProgress(...) / contentDescription = ... }` calls — tracked separately under issues.md GenericJam#11 (Compose semantics).
pshoukry
pushed a commit
to pshoukry/mob
that referenced
this pull request
Sep 9, 2026
…d-around GenericJam#7 + GenericJam#8 — `MobSlider` and `MobToggle` in MobRootView.swift gained the SwiftUI AX modifiers SwiftUI doesn't apply by default (.accessibilityAdjustableAction with default step (max-min)/10 for the slider; .accessibilityLabel(label) for the toggle). Mob.Test end-to-end driving works after these. GenericJam#14 — defensive fallback in MobDev.Connector. When the primary node name times out the connector now also tries the alternate `<app>_ios@127.0.0.1` form (without the udid suffix). The connected Device.node is updated to whichever responded so downstream RPC calls use the correct address. The root cause (mob_beam.m sometimes not seeing SIMULATOR_UDID) is captured in the resolution note for a future investigation. Remaining: GenericJam#3 (WebSocket→longpoll, investigation), GenericJam#6 (16 KB page alignment, OTP rebuild), GenericJam#9 (Alert OK button, UIKit deep dive), GenericJam#10 (Android 17 SELinux, OTP source patch), GenericJam#11 (Android Compose semantics walker, ~200 lines Kotlin).
pshoukry
pushed a commit
to pshoukry/mob
that referenced
this pull request
Sep 9, 2026
…tion camera: rotate session to portrait so YOLO sees an upright scene
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
iOS's camera sensor captures in landscape-right by default. With the phone held in portrait, both
AVCaptureVideoPreviewLayerand the newAVCaptureVideoDataOutputdelivered sideways frames — invisible to the user when only the preview was used, but devastating once we started feeding pixels to an ML model trained on upright COCO images. A jar held vertically in the UI arrived at YOLO as a horizontal bar and got classified as "laptop" or "cell phone" at low confidence.Pin both the preview and the frame stream to 90° (
videoRotationAngleon iOS 17+,videoOrientation = .portraiton older builds). With this in place, the same jar lands as cup 96% — high enough that the demo no longer needs a tuned-down confidence threshold to surface anything.What you see on the preview is now what the model sees, and detection boxes align with their objects.
Test plan
mix test— 741/741xcrun clang-format --dry-run -Werror ios/mob_nif.m— cleanswiftlint ios/— only the pre-existing force_cast warningcup 96%, laptop93%, mouse64%with defaultconf_threshold: 0.25🤖 Generated with Claude Code