Mob.Camera: live frame stream API + shared AVCaptureSession - #10
Merged
Conversation
…nest
Closes the "permissions trap" surfaced by an end user trying to wire
Mob.Location into a screen and watching the iOS dialog never appear.
Two pieces:
## 1. `guides/permissions.md` — single source of truth
New extras-guide that consolidates everything OS-permission-adjacent
into one place that all the per-capability moduledocs and
`device_capabilities.md` now point at:
* Per-capability table: what `Mob.Permissions` capability maps to
which `Info.plist` key on iOS and which `AndroidManifest.xml`
`uses-permission` line on Android. Plus the operations that
need a plist key WITHOUT going through `Mob.Permissions.request/2`
(storage_save_to_photo_library, camera preview, …).
* "What `mix mob.new` ships by default" section — the template
covers camera + microphone on iOS and most capabilities on
Android, so users hit the missing-plist-key trap when they
*add* a feature post-`mob.new`. The guide names the most-common
missing keys (location, photo library, photo library add) and
pastes the snippet to drop into Info.plist.
* iOS-specific notes section covering the
not-determined-→-no-plist-key silent failure, the previously-
undocumented "what counts as :granted" for `:photo_library`
(Limited counts), notifications.
* Android-specific notes: foreground-vs-background location,
notifications on API ≤32 (no permission needed), storage and
photos on API 33+ (READ_MEDIA_* replaces READ_EXTERNAL_STORAGE).
* "Re-requesting after denial" — OS won't re-prompt; need to send
the user to Settings.
* "Diagnosing a stuck request" 5-step checklist for the exact
failure mode that motivated this guide.
* Cross-platform pattern at the end so a reader doesn't have to
leave the guide for working code.
`guides/device_capabilities.md`'s `## Permissions` blockquote and
the moduledocs for `Mob.Permissions`, `Mob.Location`, `Mob.Camera`,
`Mob.Audio`, `Mob.Photos`, `Mob.Notify` all now point readers here
on the first failure-mode they're likely to hit.
## 2. Make iOS `:location` honest
`nif_request_permission("location")` no longer synthesises
`{:permission, :location, :granted}` unconditionally. Instead it
drives a dedicated `CLLocationManager` + `MobLocationPermissionDelegate`
through `requestWhenInUseAuthorization`, reads
`locationManagerDidChangeAuthorization:`, and reports the user's
real choice (`AuthorizedWhenInUse|Always` → `:granted`,
`Denied|Restricted` → `:denied`, `NotDetermined` → keep waiting).
Knock-on improvements:
* `MobLocationDelegate` (the existing updates-delivery class) also
learns `locationManagerDidChangeAuthorization:` and dispatches
`{:location, :error, :permission_denied}` when the user revokes
mid-session or denies a `Mob.Location.get_once/1` that skipped
the explicit `request/2` step. Before this commit, that path
just stopped delivering fix events with no diagnostic — screens
sat at "waiting for fix…" indefinitely.
* `Mob.Location` moduledoc now documents both `:permission_denied`
and `:unavailable` as expected `{:location, :error, reason}`
atoms.
The new delegate is iOS 14+ only (`locationManagerDidChangeAuthorization:`,
not the deprecated `didChangeAuthorizationStatus:`); Mob's
minimum-deployment is iOS 17, so this is well within scope.
## Verified
`mix test` clean (733 tests, 0 failures, including the 10
`Mob.VendorUsbTest` we already had landing). `mix docs` emits no
warnings on the new `guides/permissions.md`. Two existing call sites
that user-screen-grade behaviour (`NifRace.LocationScreen` in the
demo, `Mob.Permissions.request(:location)` in any project) work
without changes — the new flow is a strict drop-in for the prior
fake-grant.New: start_frame_stream/2 + stop_frame_stream/1 deliver per-frame
{:camera, :frame, %{bytes, width, height, format, timestamp_ms, dropped}}
messages to the calling process. Defaults to 640×640 rgb_f32 for direct
hand-off to Nx tensors; opts let callers pick width/height/format/facing
and a software throttle (throttle_ms).
iOS implementation uses one shared AVCaptureSession (g_preview_session)
for both preview and frame stream. iOS allows only one session per
physical camera, so previous two-session design silently dropped frames.
A serial config queue (g_camera_queue) serializes input/output
attachment so start_preview + start_frame_stream compose in any order.
vImageScale_ARGB8888 handles resize + center-crop on the capture queue
before the BGRA→RGB f32 conversion. Frame bytes flow over enif_send to
the caller pid; the delegate is held in g_frame_delegate (Apple's API
does not retain it).
Android: stub returns :unsupported so callers don't crash. Live frames
on Android will land in a follow-up.
Tests: frame_stream_opts/1 covers defaults, overrides, string-keys, and
JSON encoding (8 tests, all green).Uh oh!
There was an error while loading. Please reload this page.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Mob.Camera.start_frame_stream/2+stop_frame_stream/1deliver per-frame messages to the calling process, opening up live ML inference (YOLO, depth, etc.) on iOS. Frames arrive as{:camera, :frame, %{bytes, width, height, format, timestamp_ms, dropped}}.AVCaptureSession(g_preview_session) for both preview and frame stream — iOS allows only one session per physical camera, so the previous two-session design silently dropped frames. A serial config queue (g_camera_queue) serializes input/output attachment sostart_previewandstart_frame_streamcompose in any order.vImageScale_ARGB8888on the capture queue, then BGRA → RGB f32 conversion. Defaults to 640×640 rgb_f32 (caller-overridable) so frame bytes hand off directly into Nx.:unsupported; live frames on Android land in a follow-up.Verified live with YOLOv8n on iPhone (NxEigen → EMLX/Metal): camera frames flowing at 30fps, throttled to 10 Hz by default, BEAM picks up the messages and runs inference + draws boxes via
Mob.Canvas.Test plan
mix test— 741 tests, 0 failures (pre-existing GenServer termination fromtest resolve_module/1 error handlingis unrelated)mix format,mix credo --strict— no new warningsmix erlfmt --check src/— cleanxcrun clang-format --dry-run -Werror ios/mob_nif.m— cleanswiftlint ios/— only pre-existing force_cast warningMob.Camera.frame_stream_opts/1(defaults, overrides, string keys, JSON encoding)idevicesyslog:captureOutputat 30fps,enif_sendsucceeds, no dropped/dead-receiver warnings🤖 Generated with Claude Code