MatAnyone2Kit

A self-contained Swift package that runs the MatAnyone2 single-object video-matting model in real time on the Apple Neural Engine — stable 30 fps on an iPhone 16 (A18). The six precompiled Core ML models ship inside the package, so you add one dependency and feed it camera frames; there's no separate model download or launch-time compile.

Same frame, split down the middle — raw camera on the left, MatAnyone2's real-time matte on the right. No green screen, no rotoscoping.

The conversion toolchain (and a writeup of every ANE/Swift optimization that made it real-time) lives in scripts/.

License: the package code is GPL-3.0, but the bundled MatAnyone2 weights are NTU S-Lab License 1.0 — non-commercial only. Using this package with the bundled weights is non-commercial. Details ↓

Install

Swift Package Manager — add it as a dependency in your Package.swift:

.package(url:"https://github.com/flowtyone/MatAnyone2Kit", from:"1.0.0")
// then add "MatAnyoneKitCoreML" to your target's dependencies

Or in Xcode: File ▸ Add Package Dependencies… and paste https://github.com/flowtyone/MatAnyone2Kit.

Requires iOS 17+ / macOS 14+ (the on-device ANE-eligibility probe uses MLComputePlan, iOS 17.4 / macOS 14.4, guarded internally).

Use

import MatAnyoneKitCoreML
// Loads the bundled, precompiled models; nil if they can't be loaded.
guardlet matte =MatAnyoneMatte()else{return}
// On each camera frame (BGRA CVPixelBuffer). The first frame with a person seeds the
// tracker via Vision; subsequent frames track from MatAnyone's own memory.
matte.matte(pixelBuffer){ frame inguardlet alpha = frame.alpha else{
// .passthrough — no matte yet (no person seeded). Show the full camera frame.
return}
// `alpha` is a single-channel CVPixelBuffer (OneComponent8, 1 = foreground) at the model
// working resolution (288×512), framed like the camera frame. Composite it however you like.
iflet timing = frame.timing {
// per-stage wall-clock (ms): timing.preprocessMs / inferenceMs / postprocessMs
}}

matte(_:completion:) runs synchronously on the calling queue (typically your camera/video output queue) and calls completion once. For best throughput, drive it back-to-back keeping only the freshest pending frame rather than once per camera tick.

Real-time best practices

These are the settings that get a stable 30 fps on an iPhone 16 (A18) in a live camera app.

Resolution. The bundled models run at a fixed 288×512 (portrait) — that's the working size tuned for 30 fps on the ANE, and you don't choose it (matte.workingWidth/workingHeight report it). So you control cost upstream, at capture: feed 720p or lower. Capturing at 4K just burns ISP and conversion time for frames the matte immediately downsizes to 288×512. Capture BGRA (kCVPixelFormatType_32BGRA), upright/portrait — the returned alpha is framed identically to the frame you pass in, so you composite it with the same aspect-fill UVs as the camera.

session.sessionPreset =.hd1280x720 // 720p is plenty; the matte works at 288×512
output.videoSettings =[kCVPixelBufferPixelFormatTypeKey asString: kCVPixelFormatType_32BGRA]
output.alwaysDiscardsLateVideoFrames =true

Capture at 60 fps, not 30. The matte's throughput quantum is the camera tick. At 30 fps (33 ms ticks) a 34 ms matte misses every tick and snaps to 15 fps. Pinning the camera to 60 fps (16.7 ms ticks) lets a sub-33 ms matte hold 30 fps, with a 20 fps floor otherwise. Pinning min == max also stops iOS from auto-dropping the rate in low light.

iflet maxRate = device.activeFormat.videoSupportedFrameRateRanges.map(\.maxFrameRate).max(),(try? device.lockForConfiguration())!=nil{letdur=CMTime(value:1, timescale:CMTimeScale(min(60.0, maxRate).rounded()))
device.activeVideoMinFrameDuration = dur
device.activeVideoMaxFrameDuration = dur
device.unlockForConfiguration()}

Drive it back-to-back, don't drop on the camera tick. Run the matte off the main thread on one serial queue, and when a frame arrives mid-pass keep only the freshest one, then process it the instant the current pass finishes. This decouples throughput from the 33 ms delivery window (a 35 ms matte streams at ~28 fps instead of collapsing to 15) while keeping latency at ~one frame:

finalclassMattePacer{privateletmatte:MatAnyoneMatteprivateletqueue=DispatchQueue(label:"matte", qos:.userInitiated)privateletlock=NSLock()privatevarbusy=falseprivatevarpending:CVPixelBuffer?varonResult:((CVPixelBuffer,MatAnyoneMatte.MatteFrame)->Void)?init(_ matte:MatAnyoneMatte){self.matte = matte }
// Call from your camera delegate on every frame.
func submit(_ frame:CVPixelBuffer){
lock.lock()if busy { pending = frame; lock.unlock(); return} // keep only the freshest
busy =true; lock.unlock()process(frame)}privatefunc process(_ frame:CVPixelBuffer){
queue.async{[self]in
matte.matte(frame){ result inonResult?(frame, result) // source frame + its matte, in sync
lock.lock(); letnext= pending; pending =nilif next ==nil{ busy =false}; lock.unlock()iflet next {process(next)} // run now, not on the next camera tick
}}}}

Composite the frame with its own matte.onResult hands you the exact source frame each matte was computed from — composite that pair so the cutout never lags the camera by a frame.

Load off the main thread. First-launch ANE specialization takes a few seconds; construct MatAnyoneMatte() on a background task and run a passthrough (full-frame camera) until it's ready, so your preview appears instantly.

Task.detached(priority:.utility){letmatte=MatAnyoneMatte(); /* swap it in when ready */ }

Mind the thermal envelope. The matte runs on the ANE, leaving the GPU free — but a heavy GPU renderer plus the matte plus the camera ISP sustained together will throttle the device before ProcessInfo.thermalState even reports fair. Watch per-stage latency, not just thermalState.

What's inside

type	role
MatAnyoneMatte	top-level facade: Vision seeding + stateful tracking, returns MatteFrame
MatAnyoneCoreMLEngine	stateful inference loop over the six models
MatAnyoneCoreML	model loading + efficient MLMultiArray ↔ [Float] conversion
MemoryBank / MemoryMath	key/affinity memory, top-k softmax readout (Accelerate, parallelized)

The lower-level engine (MatAnyoneCoreMLEngine, MatAnyoneCoreML) is public if you want to drive the pipeline directly; most callers only need MatAnyoneMatte.

Configuration

MatAnyoneMatte.diagnostics =true // verbose per-frame logging + ANE-eligibility probe
MatAnyoneMatte.defaultUnit =.cpuAndNeuralEngine
MatAnyoneMatte.unitOverrides =["objsummary":.cpuAndGPU] // per-model compute placement
// Load models from a custom directory instead of the bundled ones:
letmatte=MatAnyoneMatte(modelsDir: someURL)

Set these before constructing MatAnyoneMatte.

License

MatAnyone2Kit ships under two licenses — see NOTICE.md for the full breakdown:

• Swift package source code — GNU GPL-3.0 (LICENSE).
• Bundled MatAnyone2 model weights (the Core ML models in Sources/MatAnyoneKitCoreML/Resources/MatAnyone/) — NTU S-Lab License 1.0, non-commercial only. These are a Core ML conversion of the MatAnyone2 weights by S-Lab, NTU; converting them does not change their license.

⚠️Using this package with the bundled weights is non-commercial only. The GPL-3.0 on the code does not grant any commercial rights to the weights. For commercial use of the weights, contact the authors (see the weights NOTICE.md).