From 79bd96a994952452704a5abbf81e96db697069ec Mon Sep 17 00:00:00 2001 From: GenericJam Date: Thu, 14 May 2026 16:04:28 -0600 Subject: [PATCH 1/2] docs(permissions): new guide + per-module cross-refs; iOS location honest MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Closes the "permissions trap" surfaced by an end user trying to wire Mob.Location into a screen and watching the iOS dialog never appear. Two pieces: ## 1. `guides/permissions.md` — single source of truth New extras-guide that consolidates everything OS-permission-adjacent into one place that all the per-capability moduledocs and `device_capabilities.md` now point at: * Per-capability table: what `Mob.Permissions` capability maps to which `Info.plist` key on iOS and which `AndroidManifest.xml` `uses-permission` line on Android. Plus the operations that need a plist key WITHOUT going through `Mob.Permissions.request/2` (storage_save_to_photo_library, camera preview, …). * "What `mix mob.new` ships by default" section — the template covers camera + microphone on iOS and most capabilities on Android, so users hit the missing-plist-key trap when they *add* a feature post-`mob.new`. The guide names the most-common missing keys (location, photo library, photo library add) and pastes the snippet to drop into Info.plist. * iOS-specific notes section covering the not-determined-→-no-plist-key silent failure, the previously- undocumented "what counts as :granted" for `:photo_library` (Limited counts), notifications. * Android-specific notes: foreground-vs-background location, notifications on API ≤32 (no permission needed), storage and photos on API 33+ (READ_MEDIA_* replaces READ_EXTERNAL_STORAGE). * "Re-requesting after denial" — OS won't re-prompt; need to send the user to Settings. * "Diagnosing a stuck request" 5-step checklist for the exact failure mode that motivated this guide. * Cross-platform pattern at the end so a reader doesn't have to leave the guide for working code. `guides/device_capabilities.md`'s `## Permissions` blockquote and the moduledocs for `Mob.Permissions`, `Mob.Location`, `Mob.Camera`, `Mob.Audio`, `Mob.Photos`, `Mob.Notify` all now point readers here on the first failure-mode they're likely to hit. ## 2. Make iOS `:location` honest `nif_request_permission("location")` no longer synthesises `{:permission, :location, :granted}` unconditionally. Instead it drives a dedicated `CLLocationManager` + `MobLocationPermissionDelegate` through `requestWhenInUseAuthorization`, reads `locationManagerDidChangeAuthorization:`, and reports the user's real choice (`AuthorizedWhenInUse|Always` → `:granted`, `Denied|Restricted` → `:denied`, `NotDetermined` → keep waiting). Knock-on improvements: * `MobLocationDelegate` (the existing updates-delivery class) also learns `locationManagerDidChangeAuthorization:` and dispatches `{:location, :error, :permission_denied}` when the user revokes mid-session or denies a `Mob.Location.get_once/1` that skipped the explicit `request/2` step. Before this commit, that path just stopped delivering fix events with no diagnostic — screens sat at "waiting for fix…" indefinitely. * `Mob.Location` moduledoc now documents both `:permission_denied` and `:unavailable` as expected `{:location, :error, reason}` atoms. The new delegate is iOS 14+ only (`locationManagerDidChangeAuthorization:`, not the deprecated `didChangeAuthorizationStatus:`); Mob's minimum-deployment is iOS 17, so this is well within scope. ## Verified `mix test` clean (733 tests, 0 failures, including the 10 `Mob.VendorUsbTest` we already had landing). `mix docs` emits no warnings on the new `guides/permissions.md`. Two existing call sites that user-screen-grade behaviour (`NifRace.LocationScreen` in the demo, `Mob.Permissions.request(:location)` in any project) work without changes — the new flow is a strict drop-in for the prior fake-grant. --- guides/device_capabilities.md | 10 ++ guides/permissions.md | 224 ++++++++++++++++++++++++++++++++++ ios/mob_nif.m | 95 +++++++++++++- lib/mob/audio.ex | 6 + lib/mob/camera.ex | 8 +- lib/mob/location.ex | 16 +++ lib/mob/notify.ex | 5 + lib/mob/permissions.ex | 7 ++ lib/mob/photos.ex | 9 +- mix.exs | 1 + 10 files changed, 375 insertions(+), 6 deletions(-) create mode 100644 guides/permissions.md diff --git a/guides/device_capabilities.md b/guides/device_capabilities.md index dee72e73..2306a217 100644 --- a/guides/device_capabilities.md +++ b/guides/device_capabilities.md @@ -25,6 +25,16 @@ end **No permission needed:** haptics, clipboard, share sheet, file picker. +> **`Mob.Permissions.request/2` is only half the picture.** Each +> permission-gated capability also needs an `Info.plist` usage +> description (iOS) and `AndroidManifest.xml` `uses-permission` entry +> (Android). The default `mix mob.new` template covers camera + +> microphone on iOS and most capabilities on Android, but leaves +> location, photo library, etc. for you to add explicitly. See +> [permissions](permissions.html) for the per-capability table, the +> iOS-specific gotchas, and a diagnostic checklist for "the dialog +> never appears". + ## Haptic feedback `Mob.Haptic.trigger/2` fires synchronously (no `handle_info` needed) and returns the socket: diff --git a/guides/permissions.md b/guides/permissions.md new file mode 100644 index 00000000..18cd614e --- /dev/null +++ b/guides/permissions.md @@ -0,0 +1,224 @@ +# Permissions + +Single source of truth for the OS-level permissions Mob exposes, the +manifest / `Info.plist` entries each one requires, and the +platform-specific gotchas that aren't covered by the runtime API alone. + +If you're hitting "the dialog never appears" or "I called the NIF and +nothing happened", this is the first place to look. + +## TL;DR + +* Call `Mob.Permissions.request(socket, :capability)` from your screen. +* The result arrives as `handle_info({:permission, :capability, :granted | :denied}, socket)`. +* iOS additionally needs the matching `NS*UsageDescription` key in `ios/Info.plist`. Without it, the dialog is silently suppressed and you get nothing — no event, no error. +* Android additionally needs the matching `uses-permission` line in `AndroidManifest.xml`. The `mob.new` template ships most of these already; if you added a feature after generating the project, double-check. + +## The per-capability table + +| `Mob.Permissions` cap | iOS `Info.plist` key | Android `uses-permission` | Notes | +|-------------------------|-----------------------------------------------------------------|-------------------------------------------------------------------------------------------|-------| +| `:camera` | `NSCameraUsageDescription` | `android.permission.CAMERA` | Required by `Mob.Camera`. `CameraPreview` *also* needs the plist key but does not call `Mob.Permissions.request/2` — request explicitly before mounting it. | +| `:microphone` | `NSMicrophoneUsageDescription` | `android.permission.RECORD_AUDIO` | Required by `Mob.Audio.start_recording/2` and by `Mob.Camera.capture_video/2`. | +| `:photo_library` | `NSPhotoLibraryUsageDescription` | API 33+: `READ_MEDIA_IMAGES` + `READ_MEDIA_VIDEO`. API ≤32: `READ_EXTERNAL_STORAGE`. | Required by `Mob.Photos.pick/2`. | +| `:location` | `NSLocationWhenInUseUsageDescription` | `ACCESS_FINE_LOCATION` (high accuracy) and/or `ACCESS_COARSE_LOCATION` (low accuracy). | See [iOS notes below](#ios-location-extras) — the dialog timing is unusual. | +| `:notifications` | (none — handled by `UNUserNotificationCenter`) | API 33+: `android.permission.POST_NOTIFICATIONS` | iOS shows the dialog the first time `request/2` runs. Android API ≤32 doesn't need a permission at all (notifications are user-controllable in Settings). | + +Capabilities that need **no runtime permission** on either platform and +do not appear in the table: + +* `Mob.Haptic`, `Mob.Clipboard`, `Mob.Share`, `Mob.Files.pick/2`, + `Mob.Toast`, `Mob.Alert`, `Mob.WebView`, `Mob.Motion`, `Mob.Biometric` + (uses biometric prompt UI but does not require a permission grant), + `Mob.Storage` (app-local paths only). + +Capabilities that need an `Info.plist` or manifest entry **without** going +through `Mob.Permissions.request/2`: + +| Operation | iOS `Info.plist` key | Android | +|-------------------------------------------------------------------|---------------------------------|---------| +| `Mob.Storage.save_to_photo_library/2` | `NSPhotoLibraryAddUsageDescription` | Same `READ_MEDIA_*` family as `:photo_library` on API 33+. | +| `Mob.Audio.play/2` (no permission) | none | none | +| `Mob.Camera.start_preview/2` (no permission for the *preview*; capture still needs `:camera`) | `NSCameraUsageDescription` | `CAMERA` | + +## What the `mob.new` template ships by default + +If you generate a fresh project with `mix mob.new`, the template emits: + +* **`ios/Info.plist`** — `NSCameraUsageDescription` and `NSMicrophoneUsageDescription`. Nothing else. +* **`android/app/src/main/AndroidManifest.xml`** — `CAMERA`, `RECORD_AUDIO`, `ACCESS_FINE_LOCATION`, `ACCESS_COARSE_LOCATION`, `READ_MEDIA_IMAGES`, `READ_MEDIA_VIDEO`, `READ_EXTERNAL_STORAGE` (API ≤32 only), `POST_NOTIFICATIONS`, `VIBRATE`, `FOREGROUND_SERVICE`, `INTERNET`, `RECEIVE_BOOT_COMPLETED`. + +So out-of-the-box your project covers camera + microphone on both +platforms, plus everything Android needs for the other capabilities. +**Anything iOS-side beyond camera + mic needs you to add the +`Info.plist` key yourself** before the first time you call that +capability. The most common ones to add: + +```xml +NSLocationWhenInUseUsageDescription +MyApp shows your location to ... + +NSPhotoLibraryUsageDescription +MyApp lets you pick photos from your library. + +NSPhotoLibraryAddUsageDescription +MyApp saves captures to your photo library. +``` + +If you ship without the key, iOS won't even log the missing-key error +in any obvious place — the dialog just silently doesn't appear, and +the underlying `request*Authorization` call no-ops. Symptom looks +identical to "the user denied permission" except no `denied` event +ever arrives. + +## iOS-specific notes + +### iOS location extras + +Apple's `CLLocationManager` couples permission and updates more +tightly than the other capabilities. Mob exposes both paths: + +1. `Mob.Permissions.request(socket, :location)` calls + `requestWhenInUseAuthorization` on a dedicated `CLLocationManager` + and reports the user's actual choice as `{:permission, :location, + :granted | :denied}` once the dialog is dismissed (or immediately + if the permission was previously decided). + +2. `Mob.Location.get_once/1` and `Mob.Location.start/2` *also* + trigger the dialog if `request/2` wasn't called yet. The dialog + is one-shot per app install — subsequent calls short-circuit + with the cached authorization. + +3. If the user denies, two events flow: + - `Mob.Permissions.request/2`'s caller hears `{:permission, + :location, :denied}`. + - `Mob.Location.get_once/1`/`start/2`'s caller hears + `{:location, :error, :permission_denied}` (via the + `locationManagerDidChangeAuthorization:` callback). This means + a screen that skipped `request/2` and went straight to + `get_once` still has a way to break out of the "waiting for + fix…" state. + +4. The `Allow Once` button on iOS counts as `:granted` for the + current run of the app. The next launch will prompt again. + +5. Authorization can change mid-session — the user pops out to + Settings and revokes. The delegate fires + `{:location, :error, :permission_denied}` when that happens; + surface it in your screen if you care about long-running tracking + sessions. + +### Camera + microphone + +These go through `AVFoundation`'s `requestAccessForMediaType`, which +fires the dialog at `request/2` time. No additional gotchas — make +sure the plist key is present, the dialog appears, you get a typed +`{:permission, :camera | :microphone, ...}` event. + +### Photo library + +`PHPhotoLibrary.requestAuthorizationForAccessLevel:PHAccessLevelReadWrite` +is what `:photo_library` invokes. iOS treats +`PHAuthorizationStatusLimited` (the user picked "Selected Photos…") +as `:granted` from your screen's perspective — the rest of `Mob.Photos` +deals with the limited-access set transparently. + +### Notifications + +Uses `UNUserNotificationCenter requestAuthorizationWithOptions:`. Asks +for alert, sound, and badge in one shot. The current implementation +returns `:granted` if the user granted any of the three. + +## Android-specific notes + +### Foreground vs background location + +Mob only requests *foreground* location (`ACCESS_FINE_LOCATION` / +`ACCESS_COARSE_LOCATION`). If your app needs to keep tracking while +backgrounded, you need to additionally declare +`ACCESS_BACKGROUND_LOCATION` in the manifest and request it through +a custom flow — `Mob.Permissions.request/2` doesn't surface that +capability today. + +### Notifications on Android ≤ 12 + +Pre-API-33, posting a notification does not require a runtime +permission grant — the user controls it via Settings. The +`{:permission, :notifications, :granted}` event will still fire from +`request/2` so your screen code stays portable. + +### Storage and photos + +API 33+ replaced the single `READ_EXTERNAL_STORAGE` permission with +per-media-type permissions (`READ_MEDIA_IMAGES`, `READ_MEDIA_VIDEO`). +The `mob.new` template declares all of them so the photo picker works +across API levels. Saving with `Mob.Storage.save_to_photo_library/2` +uses `MediaStore`, which doesn't require a permission on API 29+ at +all — the manifest declarations are only for the read path. + +## Re-requesting after denial + +Calling `Mob.Permissions.request/2` again *after* the user denied +does **not** re-show the dialog on either platform — that's an OS +restriction. The event still arrives (with `:denied`), so your screen +can re-render an explanation. To actually re-prompt the user, they +have to go through system Settings: + +* iOS: Settings → MyApp → \ +* Android: Settings → Apps → MyApp → Permissions → \ + +A common UX is: on `:denied`, show a "Permission needed — open +Settings" CTA. `Mob.OpenUrl.open/2` with the appropriate scheme +(`"app-settings:"` on iOS, `Intent.ACTION_APPLICATION_DETAILS_SETTINGS` +on Android — surfaced via `Mob.System.open_app_settings/1` if your +project has it; otherwise call the manifest-permitted scheme directly) +will jump straight to the right settings page. + +## Diagnosing a stuck request + +Symptom: you called `Mob.Permissions.request/2` (or a capability +function), no dialog appears, no `:permission`/`:error` event ever +arrives. + +Run through this in order: + +1. **iOS plist key present?** Open `ios/Info.plist` (or the rendered + bundle inside the `.app`) and confirm the `NS*UsageDescription` + for the capability is there. The single most common cause. +2. **Android manifest entry present?** Open + `android/app/src/main/AndroidManifest.xml`. If you added the + feature post-`mob.new`, the entry may be missing. +3. **Already denied at the OS level?** iOS: Settings → + MyApp → \. Android: Settings → Apps → MyApp → + Permissions. A previously-denied permission won't re-prompt; + `request/2` still fires the `:denied` event, so check your + `handle_info({:permission, :cap, :denied}, _)` clause exists. +4. **The screen process actually still alive?** If your screen + crashed before `handle_info/2` ran, the message is lost. Check + `adb logcat` or the iOS device console for a crash earlier in the + pipeline. +5. **You're calling `request/2` from a non-screen process.** + `enif_send` targets the calling pid; if a Task or `spawn` ran the + request, its inbox is where the event went. Always request from + the screen GenServer. + +## Cross-platform pattern + +```elixir +def mount(_params, _session, socket) do + # Cheap and idempotent on both platforms. Safe to call even if + # you're not yet ready to use the capability — the response + # informs whether the action button below should be enabled. + socket = Mob.Permissions.request(socket, :location) + {:ok, Mob.Socket.assign(socket, permission: :pending)} +end + +def handle_info({:permission, :location, :granted}, socket) do + {:noreply, Mob.Socket.assign(socket, permission: :granted)} +end + +def handle_info({:permission, :location, :denied}, socket) do + # Render a "needs permission — open Settings" CTA. + {:noreply, Mob.Socket.assign(socket, permission: :denied)} +end +``` diff --git a/ios/mob_nif.m b/ios/mob_nif.m index c9a46c3d..4e9ddc87 100644 --- a/ios/mob_nif.m +++ b/ios/mob_nif.m @@ -2089,9 +2089,14 @@ static ERL_NIF_TERM nif_request_permission(ErlNifEnv *env, int argc, const ERL_N ok ? "granted" : "denied"); }]; } else if (strcmp(cap, "location") == 0) { - // Location permission is requested via CLLocationManager when get_once/start are called. - // Here we just signal granted for iOS (the actual dialog shows at location call time). - mob_send3(&pid, "permission", "location", "granted"); + // Honest location-permission flow: drive CLLocationManager + // directly and let its delegate report the user's actual + // choice. See request_location_permission/1 below for the + // delegate setup. Previously this branch synthesised + // `:granted` unconditionally — that lied about denials, hid + // the "not determined → no plist key" failure mode, and made + // Mob.Permissions behave differently on iOS vs Android. + request_location_permission(pid); } else if (strcmp(cap, "notifications") == 0) { UNUserNotificationCenter *center = [UNUserNotificationCenter currentNotificationCenter]; [center @@ -2137,6 +2142,67 @@ static ERL_NIF_TERM nif_biometric_authenticate(ErlNifEnv *env, int argc, return enif_make_atom(env, "ok"); } +// ── Location permission ─────────────────────────────────────────────────── +// +// Separated from the location-updates delegate below so a screen can +// request authorization without also starting (and paying for) GPS +// updates. The delegate fires once per real-user choice; we map +// AuthorizedWhenInUse / AuthorizedAlways → :granted, Denied / +// Restricted → :denied. NotDetermined is the transient state before +// the dialog has been answered — we keep the delegate alive (static +// strong refs) so iOS can call back into it when the answer arrives. + +@interface MobLocationPermissionDelegate : NSObject +@property(nonatomic) ErlNifPid pid; +@property(nonatomic) BOOL resolved; +@end + +static MobLocationPermissionDelegate *g_permission_delegate = nil; +static CLLocationManager *g_permission_manager = nil; + +@implementation MobLocationPermissionDelegate +// `locationManagerDidChangeAuthorization:` is the iOS 14+ replacement +// for `locationManager:didChangeAuthorizationStatus:`. Mob targets +// iOS 17+ (see ios/build_device.zig minimum-deployment) so the older +// callback is omitted. +- (void)locationManagerDidChangeAuthorization:(CLLocationManager *)manager { + CLAuthorizationStatus status = manager.authorizationStatus; + if (status == kCLAuthorizationStatusNotDetermined) { + // Dialog is still on screen / the OS hasn't picked an initial + // state. We'll be called again with the user's choice. + return; + } + if (self.resolved) { + // Subsequent authorization changes (user revokes/grants via + // Settings) — fire the event again so the screen can react. + // Marked here for symmetry, no early return. + } + self.resolved = YES; + + ErlNifPid p = self.pid; + BOOL granted = (status == kCLAuthorizationStatusAuthorizedWhenInUse || + status == kCLAuthorizationStatusAuthorizedAlways); + mob_send3(&p, "permission", "location", granted ? "granted" : "denied"); +} +@end + +static void request_location_permission(ErlNifPid pid) { + dispatch_async(dispatch_get_main_queue(), ^{ + if (!g_permission_manager) { + g_permission_manager = [[CLLocationManager alloc] init]; + } + g_permission_delegate = [[MobLocationPermissionDelegate alloc] init]; + g_permission_delegate.pid = pid; + g_permission_manager.delegate = g_permission_delegate; + // Reading `authorizationStatus` here would tell us if we should + // skip the request entirely, but doing so synchronously can + // momentarily return NotDetermined on first launch. Calling + // `requestWhenInUseAuthorization` is idempotent — already-granted + // permissions short-circuit and the delegate fires immediately. + [g_permission_manager requestWhenInUseAuthorization]; + }); +} + // ── Location ────────────────────────────────────────────────────────────── @interface MobLocationDelegate : NSObject @@ -2181,6 +2247,29 @@ - (void)locationManager:(CLLocationManager *)mgr didFailWithError:(NSError *)err enif_send(NULL, &p, e, msg); enif_free_env(e); } +// Surface authorization-state changes through the same delegate so a +// screen that called `Mob.Location.get_once/1` without first going +// through `Mob.Permissions.request/2` still hears about denial — +// without this, `didFailWithError` doesn't fire on denial and the +// screen sits at "waiting for fix…" forever. The permission-only +// delegate above sends `{:permission, :location, ...}`; here we send +// `{:location, :error, :permission_denied}` so the screen's +// `handle_info({:location, :error, _}, _)` path catches it. +- (void)locationManagerDidChangeAuthorization:(CLLocationManager *)mgr { + CLAuthorizationStatus status = mgr.authorizationStatus; + if (status == kCLAuthorizationStatusNotDetermined) return; + if (status == kCLAuthorizationStatusAuthorizedWhenInUse || + status == kCLAuthorizationStatusAuthorizedAlways) { + return; + } + ErlNifPid p = self.pid; + ErlNifEnv *e = enif_alloc_env(); + ERL_NIF_TERM msg = + enif_make_tuple3(e, enif_make_atom(e, "location"), enif_make_atom(e, "error"), + enif_make_atom(e, "permission_denied")); + enif_send(NULL, &p, e, msg); + enif_free_env(e); +} @end static void setup_location_manager(ErlNifPid pid, BOOL oneShot, NSString *accuracy) { diff --git a/lib/mob/audio.ex b/lib/mob/audio.ex index 2b180ce3..bcdc1637 100644 --- a/lib/mob/audio.ex +++ b/lib/mob/audio.ex @@ -3,6 +3,12 @@ defmodule Mob.Audio do Microphone recording and audio playback. Recording requires `:microphone` permission (`Mob.Permissions.request/2`). + iOS additionally needs `NSMicrophoneUsageDescription` in + `Info.plist`; Android needs `RECORD_AUDIO` in + `AndroidManifest.xml`. The default `mix mob.new` templates ship + both. See the [permissions guide](permissions.html) for the + cross-platform table. + Playback requires no permission. ## Recording diff --git a/lib/mob/camera.ex b/lib/mob/camera.ex index 3d3ad4d2..706aaffb 100644 --- a/lib/mob/camera.ex +++ b/lib/mob/camera.ex @@ -2,7 +2,13 @@ defmodule Mob.Camera do @moduledoc """ Native camera capture for photos and videos. - Requires `:camera` permission (and `:microphone` for video). + Requires `:camera` permission (and `:microphone` for video). iOS + additionally needs `NSCameraUsageDescription` (and + `NSMicrophoneUsageDescription` for video) in `Info.plist`; + Android needs `CAMERA` (and `RECORD_AUDIO` for video) in + `AndroidManifest.xml`. The default `mix mob.new` templates ship + both. See the [permissions guide](permissions.html) for the + cross-platform table. Opens the native OS camera UI. Results arrive as: diff --git a/lib/mob/location.ex b/lib/mob/location.ex index b8f3c347..c7f2201e 100644 --- a/lib/mob/location.ex +++ b/lib/mob/location.ex @@ -3,12 +3,28 @@ defmodule Mob.Location do Device location (GPS / network). Requires `:location` permission (request via `Mob.Permissions.request/2`). + iOS additionally needs `NSLocationWhenInUseUsageDescription` in + `Info.plist`; Android needs `ACCESS_FINE_LOCATION` and/or + `ACCESS_COARSE_LOCATION` in `AndroidManifest.xml`. See the + [permissions guide](permissions.html) for the cross-platform table + and the "the dialog never appears" failure mode — a missing plist + key or manifest entry is the single most common reason this module + silently does nothing. Location updates arrive as: handle_info({:location, %{lat: lat, lon: lon, accuracy: acc, altitude: alt}}, socket) handle_info({:location, :error, reason}, socket) + Common `reason` atoms: + + * `:permission_denied` — user denied `:location` (or revoked it + mid-session via Settings). iOS surfaces this through + `locationManagerDidChangeAuthorization:`; Android via the + permission flow. + * `:unavailable` — the OS can't get a fix right now + (`CLLocationManager.didFailWithError`). + iOS: `CLLocationManager`. Android: `FusedLocationProviderClient`. """ diff --git a/lib/mob/notify.ex b/lib/mob/notify.ex index 9b8c3d39..2dbb3bcb 100644 --- a/lib/mob/notify.ex +++ b/lib/mob/notify.ex @@ -3,6 +3,11 @@ defmodule Mob.Notify do Local and push notifications. Requires `:notifications` permission (request via `Mob.Permissions.request/2`). + No `Info.plist` key needed on iOS. Android 13+ (API 33) requires + `POST_NOTIFICATIONS` in `AndroidManifest.xml`; older Android + versions are user-controlled via system settings. The default + `mix mob.new` template ships `POST_NOTIFICATIONS`. See the + [permissions guide](permissions.html) for the cross-platform table. All notifications arrive via `handle_info` regardless of app state (foreground, background, or relaunched after being killed). No special `mount/3` handling needed. diff --git a/lib/mob/permissions.ex b/lib/mob/permissions.ex index d1fe24cd..b3d74336 100644 --- a/lib/mob/permissions.ex +++ b/lib/mob/permissions.ex @@ -14,6 +14,13 @@ defmodule Mob.Permissions do - `:notifications` Capabilities that need *no* permission: haptics, clipboard, share sheet, file picker. + + > **Beyond `request/2`**: each capability also needs a matching + > `Info.plist` key (iOS) and `AndroidManifest.xml` entry. Without + > them the dialog is silently suppressed and you get no event. See + > the [permissions guide](permissions.html) for the per-capability + > table and the most common failure modes — it's the first place + > to check when "the dialog never appears". """ @type capability :: :camera | :microphone | :photo_library | :location | :notifications diff --git a/lib/mob/photos.ex b/lib/mob/photos.ex index eb869e93..02e39792 100644 --- a/lib/mob/photos.ex +++ b/lib/mob/photos.ex @@ -2,8 +2,13 @@ defmodule Mob.Photos do @moduledoc """ Photo / video library picker. - On iOS 14+ no permission is required (the picker itself is sandboxed). - On Android, `READ_MEDIA_IMAGES` / `READ_MEDIA_VIDEO` may be needed. + On iOS 14+ no permission is required for the picker (it runs out of + process). `Mob.Storage.save_to_photo_library/2` does require + `NSPhotoLibraryAddUsageDescription` in `Info.plist`. On Android, + `READ_MEDIA_IMAGES` / `READ_MEDIA_VIDEO` (API 33+) or + `READ_EXTERNAL_STORAGE` (API ≤ 32) need to be declared in + `AndroidManifest.xml` — `mix mob.new` ships all three. See the + [permissions guide](permissions.html) for the cross-platform table. Results arrive as: diff --git a/mix.exs b/mix.exs index 24939d65..862c6a84 100644 --- a/mix.exs +++ b/mix.exs @@ -75,6 +75,7 @@ defmodule Mob.MixProject do "guides/theming.md": [title: "Theming"], "guides/navigation.md": [title: "Navigation"], "guides/device_capabilities.md": [title: "Device Capabilities"], + "guides/permissions.md": [title: "Permissions"], "guides/native_extensions.md": [title: "Native Extensions (NIFs, features)"], "guides/dns_on_ios.md": [title: "DNS on iOS"], "guides/push_notifications.md": [title: "Push Notifications"], From 7d3f315f8ff057606394382b7cb689227091b482 Mon Sep 17 00:00:00 2001 From: GenericJam Date: Fri, 15 May 2026 10:00:46 -0600 Subject: [PATCH 2/2] Mob.Camera: live frame stream + shared AVCaptureSession MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit New: start_frame_stream/2 + stop_frame_stream/1 deliver per-frame {:camera, :frame, %{bytes, width, height, format, timestamp_ms, dropped}} messages to the calling process. Defaults to 640×640 rgb_f32 for direct hand-off to Nx tensors; opts let callers pick width/height/format/facing and a software throttle (throttle_ms). iOS implementation uses one shared AVCaptureSession (g_preview_session) for both preview and frame stream. iOS allows only one session per physical camera, so previous two-session design silently dropped frames. A serial config queue (g_camera_queue) serializes input/output attachment so start_preview + start_frame_stream compose in any order. vImageScale_ARGB8888 handles resize + center-crop on the capture queue before the BGRA→RGB f32 conversion. Frame bytes flow over enif_send to the caller pid; the delegate is held in g_frame_delegate (Apple's API does not retain it). Android: stub returns :unsupported so callers don't crash. Live frames on Android will land in a follow-up. Tests: frame_stream_opts/1 covers defaults, overrides, string-keys, and JSON encoding (8 tests, all green). --- android/jni/mob_nif.zig | 26 +++ ios/mob_nif.m | 395 ++++++++++++++++++++++++++++++++++++--- lib/mob/camera.ex | 94 ++++++++++ src/mob_nif.erl | 6 + test/mob/camera_test.exs | 82 ++++++++ 5 files changed, 578 insertions(+), 25 deletions(-) create mode 100644 test/mob/camera_test.exs diff --git a/android/jni/mob_nif.zig b/android/jni/mob_nif.zig index ac12cab6..e65549fd 100644 --- a/android/jni/mob_nif.zig +++ b/android/jni/mob_nif.zig @@ -2326,6 +2326,30 @@ export fn nif_camera_stop_preview( return erts.ok(env); } +// Live camera frame stream — Android implementation pending (needs +// Camera2 + ImageAnalysis wiring on the Kotlin side). Returns +// :unsupported for now so the iOS demo unblocks without breaking the +// Android build. Track in https://github.com/GenericJam/mob/issues +export fn nif_camera_start_frame_stream( + env: ?*erts.ErlNifEnv, + argc: c_int, + argv: [*]const erts.ERL_NIF_TERM, +) callconv(.c) erts.ERL_NIF_TERM { + _ = argc; + _ = argv; + return erts.atom(env, "unsupported"); +} + +export fn nif_camera_stop_frame_stream( + env: ?*erts.ErlNifEnv, + argc: c_int, + argv: [*]const erts.ERL_NIF_TERM, +) callconv(.c) erts.ERL_NIF_TERM { + _ = argc; + _ = argv; + return erts.atom(env, "unsupported"); +} + export fn nif_photos_pick( env: ?*erts.ErlNifEnv, argc: c_int, @@ -3266,6 +3290,8 @@ const nif_funcs = [_]erts.ErlNifFunc{ .{ .name = "camera_capture_video", .arity = 1, .fptr = nif_camera_capture_video, .flags = 0 }, .{ .name = "camera_start_preview", .arity = 1, .fptr = nif_camera_start_preview, .flags = 0 }, .{ .name = "camera_stop_preview", .arity = 0, .fptr = nif_camera_stop_preview, .flags = 0 }, + .{ .name = "camera_start_frame_stream", .arity = 1, .fptr = nif_camera_start_frame_stream, .flags = 0 }, + .{ .name = "camera_stop_frame_stream", .arity = 0, .fptr = nif_camera_stop_frame_stream, .flags = 0 }, .{ .name = "photos_pick", .arity = 2, .fptr = nif_photos_pick, .flags = 0 }, .{ .name = "files_pick", .arity = 1, .fptr = nif_files_pick, .flags = 0 }, .{ .name = "audio_start_recording", .arity = 1, .fptr = nif_audio_start_recording, .flags = 0 }, diff --git a/ios/mob_nif.m b/ios/mob_nif.m index 4e9ddc87..e1df8bf0 100644 --- a/ios/mob_nif.m +++ b/ios/mob_nif.m @@ -31,6 +31,7 @@ #import "MobNode.h" #include "erl_nif.h" #import +#import #import #import #import @@ -2064,6 +2065,11 @@ static ERL_NIF_TERM nif_take_launch_notification(ErlNifEnv *env, int argc, // ── Permission request ──────────────────────────────────────────────────── +// Forward declaration — definition lives below the +// `MobLocationPermissionDelegate` class so it can reference its +// instance methods. C99 forbids implicit function declarations. +static void request_location_permission(ErlNifPid pid); + static ERL_NIF_TERM nif_request_permission(ErlNifEnv *env, int argc, const ERL_NIF_TERM argv[]) { char cap[32]; if (!enif_get_atom(env, argv[0], cap, sizeof(cap), ERL_NIF_LATIN1)) @@ -2257,7 +2263,8 @@ - (void)locationManager:(CLLocationManager *)mgr didFailWithError:(NSError *)err // `handle_info({:location, :error, _}, _)` path catches it. - (void)locationManagerDidChangeAuthorization:(CLLocationManager *)mgr { CLAuthorizationStatus status = mgr.authorizationStatus; - if (status == kCLAuthorizationStatusNotDetermined) return; + if (status == kCLAuthorizationStatusNotDetermined) + return; if (status == kCLAuthorizationStatusAuthorizedWhenInUse || status == kCLAuthorizationStatusAuthorizedAlways) { return; @@ -2424,7 +2431,74 @@ static ERL_NIF_TERM nif_camera_capture_video(ErlNifEnv *env, int argc, const ERL // ── Camera preview ──────────────────────────────────────────────────────── -AVCaptureSession *g_preview_session = nil; +// One shared AVCaptureSession per app — iOS won't allow two sessions on +// the same physical camera, and a single session can carry multiple +// outputs (preview layer + AVCaptureVideoDataOutput). Both +// `start_preview` and `start_frame_stream` configure this same session; +// the serial queue serializes all mutation so they can be called in +// either order. +AVCaptureSession *g_preview_session = nil; // exported name preserved for SwiftUI +static AVCaptureDeviceInput *g_camera_input = nil; +static NSString *g_camera_facing = nil; +static dispatch_queue_t g_camera_queue = NULL; + +static dispatch_queue_t mob_camera_queue(void) { + static dispatch_once_t once; + dispatch_once(&once, ^{ + g_camera_queue = dispatch_queue_create("io.mob.camera.config", DISPATCH_QUEUE_SERIAL); + }); + return g_camera_queue; +} + +// Configure session input for the requested facing. Idempotent — if the +// facing already matches, leaves the input alone. Must be called from +// the serial camera queue. +static BOOL mob_camera_ensure_session(NSString *facing) { + if (!g_preview_session) { + g_preview_session = [[AVCaptureSession alloc] init]; + g_preview_session.sessionPreset = AVCaptureSessionPresetHigh; + NSLog(@"[mob/camera] created shared AVCaptureSession"); + } + if (g_camera_input && [g_camera_facing isEqualToString:facing]) { + return YES; + } + + AVCaptureDevicePosition position = [facing isEqualToString:@"front"] + ? AVCaptureDevicePositionFront + : AVCaptureDevicePositionBack; + AVCaptureDevice *device = + [AVCaptureDevice defaultDeviceWithDeviceType:AVCaptureDeviceTypeBuiltInWideAngleCamera + mediaType:AVMediaTypeVideo + position:position]; + if (!device) { + NSLog(@"[mob/camera] no camera device for facing=%@", facing); + return NO; + } + NSError *err = nil; + AVCaptureDeviceInput *input = [AVCaptureDeviceInput deviceInputWithDevice:device error:&err]; + if (!input) { + NSLog(@"[mob/camera] AVCaptureDeviceInput failed: %@", err); + return NO; + } + + [g_preview_session beginConfiguration]; + if (g_camera_input) { + [g_preview_session removeInput:g_camera_input]; + g_camera_input = nil; + } + if ([g_preview_session canAddInput:input]) { + [g_preview_session addInput:input]; + g_camera_input = input; + g_camera_facing = [facing copy]; + NSLog(@"[mob/camera] added input facing=%@", facing); + } else { + NSLog(@"[mob/camera] canAddInput=NO for facing=%@", facing); + [g_preview_session commitConfiguration]; + return NO; + } + [g_preview_session commitConfiguration]; + return YES; +} static ERL_NIF_TERM nif_camera_start_preview(ErlNifEnv *env, int argc, const ERL_NIF_TERM argv[]) { ErlNifBinary bin; @@ -2442,31 +2516,14 @@ static ERL_NIF_TERM nif_camera_start_preview(ErlNifEnv *env, int argc, const ERL facing = @"front"; } - // Session setup and startRunning must run on a background queue (Apple requirement). - // After the session is running, update the shared global and notify the preview view - // on the main queue so SwiftUI can safely read g_preview_session. - dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0), ^{ - AVCaptureDevicePosition position = [facing isEqualToString:@"front"] - ? AVCaptureDevicePositionFront - : AVCaptureDevicePositionBack; - AVCaptureDevice *device = - [AVCaptureDevice defaultDeviceWithDeviceType:AVCaptureDeviceTypeBuiltInWideAngleCamera - mediaType:AVMediaTypeVideo - position:position]; - if (!device) - return; - AVCaptureDeviceInput *input = [AVCaptureDeviceInput deviceInputWithDevice:device error:nil]; - if (!input) + dispatch_async(mob_camera_queue(), ^{ + if (!mob_camera_ensure_session(facing)) return; - AVCaptureSession *session = [[AVCaptureSession alloc] init]; - session.sessionPreset = AVCaptureSessionPresetHigh; - if ([session canAddInput:input]) - [session addInput:input]; - [session startRunning]; + if (!g_preview_session.isRunning) { + [g_preview_session startRunning]; + NSLog(@"[mob/camera] session startRunning (from preview)"); + } dispatch_async(dispatch_get_main_queue(), ^{ - if (g_preview_session) - [g_preview_session stopRunning]; - g_preview_session = session; [[NSNotificationCenter defaultCenter] postNotificationName:@"MobCameraSessionChanged" object:nil]; }); @@ -2489,6 +2546,292 @@ static ERL_NIF_TERM nif_camera_stop_preview(ErlNifEnv *env, int argc, const ERL_ return enif_make_atom(env, "ok"); } +// ── Camera frame stream ─────────────────────────────────────────────────── +// Delivers per-frame pixel data to a BEAM process as messages of shape: +// +// {camera, frame, #{bytes, width, height, format, timestamp_ms, dropped}} +// +// The capture session here is independent of the preview session — they +// each own their own AVCaptureDeviceInput on the same camera device (an +// arrangement AVFoundation allows). This means start_frame_stream can run +// headlessly (no visible preview) and start_preview can run without ML +// inference. The two compose cleanly when both are active. +// +// vImage handles resize + format conversion on the capture queue so the +// BEAM mailbox never sees raw camera buffers. Late frames are discarded +// at the AVFoundation layer (alwaysDiscardsLateVideoFrames=YES is the +// iOS default and we leave it on); throttle_ms adds an additional +// software gate when callers want a slower delivery rate than the +// camera's native 30fps. + +@interface MobFrameDelegate : NSObject +@end + +@implementation MobFrameDelegate { + ErlNifPid _receiver_pid; + int _target_width; + int _target_height; + NSString *_format; + int _throttle_ms; + uint64_t _last_delivered_ms; + uint64_t _dropped_count; +} + +- (instancetype)initWithPid:(ErlNifPid)pid + width:(int)width + height:(int)height + format:(NSString *)format + throttleMs:(int)throttleMs { + if ((self = [super init])) { + _receiver_pid = pid; + _target_width = width; + _target_height = height; + _format = format; + _throttle_ms = throttleMs; + _last_delivered_ms = 0; + _dropped_count = 0; + } + return self; +} + +- (void)captureOutput:(AVCaptureOutput *)output + didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer + fromConnection:(AVCaptureConnection *)connection { + uint64_t now_ms = (uint64_t)([[NSDate date] timeIntervalSince1970] * 1000.0); + + // Software-side throttle: gate at most one frame per `throttle_ms`. + // Frames arriving faster get counted in `dropped` for visibility. + if (_throttle_ms > 0 && (now_ms - _last_delivered_ms) < (uint64_t)_throttle_ms) { + _dropped_count++; + return; + } + + CVPixelBufferRef pixbuf = CMSampleBufferGetImageBuffer(sampleBuffer); + if (!pixbuf) { + _dropped_count++; + return; + } + + CVPixelBufferLockBaseAddress(pixbuf, kCVPixelBufferLock_ReadOnly); + + size_t src_w = CVPixelBufferGetWidth(pixbuf); + size_t src_h = CVPixelBufferGetHeight(pixbuf); + void *src_base = CVPixelBufferGetBaseAddress(pixbuf); + size_t src_stride = CVPixelBufferGetBytesPerRow(pixbuf); + + // Center-crop the source to the destination aspect ratio so the + // resize doesn't squash a 16:9 camera frame into a 1:1 tensor. + // For a 1920×1080 source and a 640×640 destination: take a centered + // 1080×1080 square (cuts 420 px from each side), then scale to + // 640×640. + int dst_w = _target_width; + int dst_h = _target_height; + + double src_aspect = (double)src_w / (double)src_h; + double dst_aspect = (double)dst_w / (double)dst_h; + + size_t crop_x = 0, crop_y = 0, crop_w = src_w, crop_h = src_h; + if (src_aspect > dst_aspect) { + // Source is wider than destination — crop horizontally. + crop_w = (size_t)((double)src_h * dst_aspect); + crop_x = (src_w - crop_w) / 2; + } else if (src_aspect < dst_aspect) { + // Source is taller than destination — crop vertically. + crop_h = (size_t)((double)src_w / dst_aspect); + crop_y = (src_h - crop_h) / 2; + } + + // vImage source descriptor pointing at the (possibly-offset) crop region. + // Bytes per pixel for BGRA = 4. + vImage_Buffer vsrc = { + .data = (uint8_t *)src_base + (crop_y * src_stride) + (crop_x * 4), + .height = crop_h, + .width = crop_w, + .rowBytes = src_stride, + }; + + // Intermediate BGRA8 destination at the target size. + uint8_t *dst_bgra = malloc((size_t)dst_w * dst_h * 4); + vImage_Buffer vdst = { + .data = dst_bgra, + .height = (vImagePixelCount)dst_h, + .width = (vImagePixelCount)dst_w, + .rowBytes = (size_t)dst_w * 4, + }; + + vImageScale_ARGB8888(&vsrc, &vdst, NULL, kvImageHighQualityResampling); + + CVPixelBufferUnlockBaseAddress(pixbuf, kCVPixelBufferLock_ReadOnly); + + // Pack into the requested output format. + ErlNifEnv *msg_env = enif_alloc_env(); + ErlNifBinary out_bin; + + if ([_format isEqualToString:@"rgb_f32"]) { + size_t pixel_count = (size_t)dst_w * (size_t)dst_h; + enif_alloc_binary(pixel_count * 3 * sizeof(float), &out_bin); + float *out = (float *)out_bin.data; + + // vImage delivers BGRA8 in iOS-native channel order. Convert to + // interleaved RGB f32 in [0, 1]. Straight loop is fine — vImage + // doesn't ship a BGRA→RGB-interleaved-f32 single-call so this + // would otherwise be three passes (BGRA→RGBA→planar→combine). + // The single-pass loop is ~1ms on a 640×640 frame. + for (size_t i = 0; i < pixel_count; i++) { + uint8_t b = dst_bgra[i * 4 + 0]; + uint8_t g = dst_bgra[i * 4 + 1]; + uint8_t r = dst_bgra[i * 4 + 2]; + out[i * 3 + 0] = (float)r / 255.0f; + out[i * 3 + 1] = (float)g / 255.0f; + out[i * 3 + 2] = (float)b / 255.0f; + } + } else { + // :bgra_u8 — copy bytes directly. + enif_alloc_binary((size_t)dst_w * dst_h * 4, &out_bin); + memcpy(out_bin.data, dst_bgra, (size_t)dst_w * dst_h * 4); + } + free(dst_bgra); + + // Build the result map. Keys are atoms so the Elixir side gets + // %{bytes:, width:, height:, format:, timestamp_ms:, dropped:}. + ERL_NIF_TERM bytes_term = enif_make_binary(msg_env, &out_bin); + ERL_NIF_TERM map = enif_make_new_map(msg_env); + enif_make_map_put(msg_env, map, enif_make_atom(msg_env, "bytes"), bytes_term, &map); + enif_make_map_put(msg_env, map, enif_make_atom(msg_env, "width"), enif_make_int(msg_env, dst_w), + &map); + enif_make_map_put(msg_env, map, enif_make_atom(msg_env, "height"), + enif_make_int(msg_env, dst_h), &map); + enif_make_map_put(msg_env, map, enif_make_atom(msg_env, "format"), + enif_make_atom(msg_env, [_format UTF8String]), &map); + enif_make_map_put(msg_env, map, enif_make_atom(msg_env, "timestamp_ms"), + enif_make_uint64(msg_env, now_ms), &map); + enif_make_map_put(msg_env, map, enif_make_atom(msg_env, "dropped"), + enif_make_uint64(msg_env, _dropped_count), &map); + + ERL_NIF_TERM tagged = enif_make_tuple3(msg_env, enif_make_atom(msg_env, "camera"), + enif_make_atom(msg_env, "frame"), map); + + // enif_send is documented thread-safe; this delegate callback runs + // on the capture session's serial queue, not the BEAM scheduler. + // Passing NULL for caller_env is the standard pattern from a + // non-BEAM thread. + enif_send(NULL, &_receiver_pid, msg_env, tagged); + enif_free_env(msg_env); + + _last_delivered_ms = now_ms; + _dropped_count = 0; +} + +@end + +// Frame stream output + delegate attach to the shared g_preview_session. +// AVCaptureVideoDataOutput does NOT retain its delegate, so g_frame_delegate +// is the canonical strong reference that keeps it alive. +static AVCaptureVideoDataOutput *g_frame_output = nil; +static MobFrameDelegate *g_frame_delegate = nil; +static dispatch_queue_t g_frame_delivery_queue = NULL; + +static ERL_NIF_TERM nif_camera_start_frame_stream(ErlNifEnv *env, int argc, + const ERL_NIF_TERM argv[]) { + ErlNifBinary bin; + if (!enif_inspect_binary(env, argv[0], &bin) && + !enif_inspect_iolist_as_binary(env, argv[0], &bin)) { + return enif_make_badarg(env); + } + + NSString *json = [[NSString alloc] initWithBytes:bin.data + length:bin.size + encoding:NSUTF8StringEncoding]; + NSDictionary *opts = + [NSJSONSerialization JSONObjectWithData:[json dataUsingEncoding:NSUTF8StringEncoding] + options:0 + error:nil]; + + int target_w = [(opts[@"width"] ?: @640) intValue]; + int target_h = [(opts[@"height"] ?: @640) intValue]; + NSString *facing = [opts[@"facing"] isEqualToString:@"front"] ? @"front" : @"back"; + NSString *format = [opts[@"format"] isEqualToString:@"bgra_u8"] ? @"bgra_u8" : @"rgb_f32"; + int throttle_ms = [(opts[@"throttle_ms"] ?: @0) intValue]; + + // Cap pixel count to keep mailbox bounded. ~4 MP = 2048×2048. + if ((int64_t)target_w * (int64_t)target_h > 4 * 1024 * 1024) { + target_w = 2048; + target_h = 2048; + } + + ErlNifPid caller_pid; + enif_self(env, &caller_pid); + + NSLog(@"[mob/camera] start_frame_stream w=%d h=%d facing=%@ format=%@ throttle=%d", target_w, + target_h, facing, format, throttle_ms); + + if (!g_frame_delivery_queue) { + g_frame_delivery_queue = + dispatch_queue_create("io.mob.camera.frame_delivery", DISPATCH_QUEUE_SERIAL); + } + + dispatch_async(mob_camera_queue(), ^{ + if (!mob_camera_ensure_session(facing)) { + NSLog(@"[mob/camera] ensure_session failed"); + return; + } + + [g_preview_session beginConfiguration]; + if (g_frame_output) { + [g_preview_session removeOutput:g_frame_output]; + g_frame_output = nil; + g_frame_delegate = nil; + } + + AVCaptureVideoDataOutput *output = [[AVCaptureVideoDataOutput alloc] init]; + output.videoSettings = @{(id)kCVPixelBufferPixelFormatTypeKey : @(kCVPixelFormatType_32BGRA)}; + output.alwaysDiscardsLateVideoFrames = YES; + + MobFrameDelegate *delegate = [[MobFrameDelegate alloc] initWithPid:caller_pid + width:target_w + height:target_h + format:format + throttleMs:throttle_ms]; + // Per Apple: setSampleBufferDelegate:queue: does NOT retain the + // delegate. Hold our own strong ref in g_frame_delegate. + [output setSampleBufferDelegate:delegate queue:g_frame_delivery_queue]; + + if ([g_preview_session canAddOutput:output]) { + [g_preview_session addOutput:output]; + g_frame_output = output; + g_frame_delegate = delegate; + NSLog(@"[mob/camera] added AVCaptureVideoDataOutput"); + } else { + NSLog(@"[mob/camera] canAddOutput=NO"); + } + [g_preview_session commitConfiguration]; + + if (!g_preview_session.isRunning) { + [g_preview_session startRunning]; + NSLog(@"[mob/camera] session startRunning (from frame_stream)"); + } else { + NSLog(@"[mob/camera] session already running"); + } + }); + + return enif_make_atom(env, "ok"); +} + +static ERL_NIF_TERM nif_camera_stop_frame_stream(ErlNifEnv *env, int argc, + const ERL_NIF_TERM argv[]) { + dispatch_async(mob_camera_queue(), ^{ + if (g_frame_output) { + [g_preview_session beginConfiguration]; + [g_preview_session removeOutput:g_frame_output]; + [g_preview_session commitConfiguration]; + g_frame_output = nil; + g_frame_delegate = nil; + NSLog(@"[mob/camera] stop_frame_stream removed output"); + } + }); + return enif_make_atom(env, "ok"); +} + // ── Photo library picker ────────────────────────────────────────────────── @interface MobPhotosDelegate : NSObject @@ -5871,6 +6214,8 @@ static ERL_NIF_TERM nif_vendor_usb_close(ErlNifEnv *env, int argc, const ERL_NIF {"camera_capture_video", 1, nif_camera_capture_video, 0}, {"camera_start_preview", 1, nif_camera_start_preview, 0}, {"camera_stop_preview", 0, nif_camera_stop_preview, 0}, + {"camera_start_frame_stream", 1, nif_camera_start_frame_stream, 0}, + {"camera_stop_frame_stream", 0, nif_camera_stop_frame_stream, 0}, {"photos_pick", 2, nif_photos_pick, 0}, {"files_pick", 1, nif_files_pick, 0}, {"audio_start_recording", 1, nif_audio_start_recording, 0}, diff --git a/lib/mob/camera.ex b/lib/mob/camera.ex index 706aaffb..96b9dddb 100644 --- a/lib/mob/camera.ex +++ b/lib/mob/camera.ex @@ -19,6 +19,20 @@ defmodule Mob.Camera do The `path` is a local temp file. Copy it elsewhere before the next capture. iOS: `UIImagePickerController`. Android: `TakePicture` / `CaptureVideo` activity contracts. + + ## Live frame stream + + For real-time work (object detection, AR, custom filters) `start_frame_stream/2` + delivers per-frame pixel data as messages: + + handle_info({:camera, :frame, %{bytes: bin, width: w, height: h, + format: :rgb_f32, + timestamp_ms: t, dropped: n}}, socket) + + The native side handles resize + format conversion (vImage on iOS) so + the BEAM never sees raw camera buffers. Late frames are dropped on + the native side — the BEAM mailbox can't unbounded-grow if your + receiver lags behind the camera's 30 fps cadence. """ @doc """ @@ -67,4 +81,84 @@ defmodule Mob.Camera do :mob_nif.camera_stop_preview() socket end + + @doc """ + Start streaming camera frames to the calling process. Frames arrive + as messages of shape: + + handle_info({:camera, :frame, %{ + bytes: binary(), # pixel data, format-dependent + width: non_neg_integer(), + height: non_neg_integer(), + format: :rgb_f32 | :bgra_u8, + timestamp_ms: non_neg_integer(), + dropped: non_neg_integer() # frames skipped since last delivery + }}, socket) + + ## Options + + * `:width`, `:height` — target frame size in pixels. Defaults to + `640` × `640` (YOLO-friendly). Pass `nil` for both to receive the + camera's native resolution. Mismatched aspect ratios are + center-cropped on the long axis before scaling. Capped at ~4 MP + to keep the BEAM mailbox bounded. + + * `:format` — pixel format. One of: + - `:rgb_f32` (default) — interleaved RGB floats normalised to + `[0.0, 1.0]`. Byte size: `width * height * 3 * 4`. Ready for + `Nx.from_binary(bin, :f32, ...) |> Nx.reshape({1, h, w, 3})`. + - `:bgra_u8` — raw 32-bit BGRA bytes, native iOS pixel layout. + Byte size: `width * height * 4`. 4× smaller than `:rgb_f32`; + useful for forwarding to another NIF or doing custom + preprocessing. + + * `:facing` — `:back` (default) or `:front`. Same camera the + preview uses; calling `start_frame_stream/2` alone will activate + the capture session without a visible preview. + + * `:throttle_ms` — minimum interval between deliveries (default + `0`). Native-side throttle, complementary to the OS's late-frame + drop. Use `throttle_ms: 100` for 10 Hz delivery when full + camera-rate inference isn't needed. + + ## Notes + + Returns the socket immediately; frames begin arriving asynchronously + once the OS has activated the capture session (typically <100 ms). + Receiver is the **calling process** at the time of invocation — + call from a `Mob.Screen` callback (mount, handle_info), not from a + task or genserver running elsewhere. + """ + @spec start_frame_stream(Mob.Socket.t(), keyword()) :: Mob.Socket.t() + def start_frame_stream(socket, opts \\ []) do + :mob_nif.camera_start_frame_stream(:json.encode(frame_stream_opts(opts))) + socket + end + + @doc """ + Build the option map passed to `camera_start_frame_stream/1`. Pure + function exposed so tests can pin defaults + serialisation without + going through the NIF. + """ + @spec frame_stream_opts(keyword()) :: map() + def frame_stream_opts(opts) do + %{ + "width" => Keyword.get(opts, :width, 640), + "height" => Keyword.get(opts, :height, 640), + "format" => Keyword.get(opts, :format, :rgb_f32) |> Atom.to_string(), + "facing" => Keyword.get(opts, :facing, :back) |> Atom.to_string(), + "throttle_ms" => Keyword.get(opts, :throttle_ms, 0) + } + end + + @doc """ + Stop the camera frame stream. Safe to call when no stream is + active. The visible preview (if `start_preview/2` was called + separately) is left untouched. + """ + @spec stop_frame_stream(Mob.Socket.t()) :: Mob.Socket.t() + def stop_frame_stream(socket) do + :mob_nif.camera_stop_frame_stream() + socket + end end diff --git a/src/mob_nif.erl b/src/mob_nif.erl index 722932d2..e5692d73 100644 --- a/src/mob_nif.erl +++ b/src/mob_nif.erl @@ -30,6 +30,8 @@ camera_capture_video/1, camera_start_preview/1, camera_stop_preview/0, + camera_start_frame_stream/1, + camera_stop_frame_stream/0, %% Photo library photos_pick/2, %% File picker @@ -133,6 +135,8 @@ camera_capture_video/1, camera_start_preview/1, camera_stop_preview/0, + camera_start_frame_stream/1, + camera_stop_frame_stream/0, photos_pick/2, files_pick/1, audio_start_recording/1, @@ -230,6 +234,8 @@ camera_capture_photo(_Quality) -> erlang:nif_error(not_loaded). camera_capture_video(_MaxDuration) -> erlang:nif_error(not_loaded). camera_start_preview(_OptsJson) -> erlang:nif_error(not_loaded). camera_stop_preview() -> erlang:nif_error(not_loaded). +camera_start_frame_stream(_OptsJson) -> erlang:nif_error(not_loaded). +camera_stop_frame_stream() -> erlang:nif_error(not_loaded). photos_pick(_Max, _Types) -> erlang:nif_error(not_loaded). files_pick(_MimeTypes) -> erlang:nif_error(not_loaded). audio_start_recording(_OptsJson) -> erlang:nif_error(not_loaded). diff --git a/test/mob/camera_test.exs b/test/mob/camera_test.exs new file mode 100644 index 00000000..59e1eeb6 --- /dev/null +++ b/test/mob/camera_test.exs @@ -0,0 +1,82 @@ +defmodule Mob.CameraTest do + use ExUnit.Case, async: true + + alias Mob.Camera + + describe "frame_stream_opts/1" do + test "defaults: 640×640 rgb_f32 back camera, no throttle" do + assert Camera.frame_stream_opts([]) == %{ + "width" => 640, + "height" => 640, + "format" => "rgb_f32", + "facing" => "back", + "throttle_ms" => 0 + } + end + + test "width / height override" do + opts = Camera.frame_stream_opts(width: 320, height: 240) + assert opts["width"] == 320 + assert opts["height"] == 240 + end + + test ":bgra_u8 format is passed through as the string \"bgra_u8\"" do + opts = Camera.frame_stream_opts(format: :bgra_u8) + assert opts["format"] == "bgra_u8" + end + + test ":front facing is passed through as the string \"front\"" do + opts = Camera.frame_stream_opts(facing: :front) + assert opts["facing"] == "front" + end + + test "throttle_ms is passed through as an integer" do + opts = Camera.frame_stream_opts(throttle_ms: 100) + assert opts["throttle_ms"] == 100 + end + + test "keys are strings, matching the rest of the NIF JSON surface" do + opts = Camera.frame_stream_opts([]) + # Audio + start_preview use string keys for their JSON-encoded + # NIF args; frame_stream_opts should follow the same convention so + # the iOS-side NSJSONSerialization deserialises a consistent shape. + for key <- ["width", "height", "format", "facing", "throttle_ms"] do + assert Map.has_key?(opts, key), "expected key #{inspect(key)}" + end + + for atom <- [:width, :height, :format, :facing, :throttle_ms] do + refute Map.has_key?(opts, atom), "found atom key #{inspect(atom)}" + end + end + + test "every option is independently overridable" do + opts = + Camera.frame_stream_opts( + width: 1280, + height: 720, + format: :bgra_u8, + facing: :front, + throttle_ms: 33 + ) + + assert opts == %{ + "width" => 1280, + "height" => 720, + "format" => "bgra_u8", + "facing" => "front", + "throttle_ms" => 33 + } + end + + test "serialises to JSON cleanly (this is what hits the NIF)" do + # The NIF receives the JSON-encoded result, so make sure the map + # round-trips through :json without losing data. Any future + # option that's not JSON-serialisable would fail here. + opts = Camera.frame_stream_opts([]) + json = :json.encode(opts) |> IO.iodata_to_binary() + + decoded = :json.decode(json) + assert decoded == opts + end + end +end