PrototypeKit

(Ironically, a prototype itself...) 😅

Status: Work In Progress

Goals 🥅

• Make it easier to prototype basic Machine Learning apps with SwiftUI
• Provide an easy interface for commonly built views to assist with prototyping and idea validation
• Effectively a wrapper around the more complex APIs, providing a simpler interface (perhaps not all the same functionality, but enough to get you started and inspired!)

Requirements 📋

• iOS 14.0+ / macOS 13.0+
• Swift 5.9+
• Xcode 15+
• Sound recognition (recognizeSounds) requires iOS 15.0+ and is unavailable on macOS

Installation 📦

PrototypeKit is distributed as a Swift Package.

Add via Xcode

1. In Xcode, choose File → Add Package Dependencies…
2. Paste the package URL into the search field:

   https://github.com/FridayTechnologies/PrototypeKit
   

3. For Dependency Rule, select Up to Next Major Version starting from 0.1.0 for reproducible builds. (To live on the latest unreleased changes instead, select Branch and enter master.)
4. Click Add Package, then add the PrototypeKit library to your app target.

Add via Package.swift

If you maintain your own Swift package, add PrototypeKit to your dependencies:

dependencies:[.package(url:"https://github.com/FridayTechnologies/PrototypeKit", from:"0.1.0")]

…then add it to your target's dependencies:

.target(
name:"YourTarget",
dependencies:["PrototypeKit"])

Note: PrototypeKit follows Semantic Versioning. It is still pre-1.0, so minor version bumps (0.x) may include breaking changes; pin with .upToNextMinor(from: "0.1.0") if you need stricter guarantees. See the CHANGELOG for what's in each release.

Documentation & examples 📚

• API reference — DocC documentation is published to GitHub Pages: https://fridaytechnologies.github.io/PrototypeKit/documentation/prototypekit
• Sample code — the Examples/ directory contains a runnable SwiftUI gallery covering every feature; copy PrototypeKitExamples.swift into an app to try them.

Examples

Here are a few basic examples you can use today.

Camera Tasks

Start Here

1. Ensure you have created your Xcode project
2. Ensure you have added the PrototypeKit package to your project (see Installation above)
3. Select your project file within the project navigator.

4. Ensure that your target is selected

5. Select the info tab.
6. Right-click within the "Custom iOS Target Properties" table, and select "Add Row"

7. Use Privacy - Camera Usage Description for the key. Type the reason your app will use the camera as the value.

Live Camera View

Utilise PKCameraView

PKCameraView()

Full Example

import SwiftUI
import PrototypeKit
structContentView:View{varbody:someView{VStack{PKCameraView()}.padding()}}

Live Image Classification

1. Required Step: Drag in your Create ML / Core ML model into Xcode.
2. Change FruitClassifier below to the name of your Model.
3. You can use latestPrediction as you would any other state variable (i.e refer to other views such as Slider)

Utilise ImageClassifierView

ImageClassifierView(modelURL:FruitClassifier.urlOfModelInThisBundle,
latestPrediction: $latestPrediction)

Full Example

import SwiftUI
import PrototypeKit
structImageClassifierViewSample:View{@StatevarlatestPrediction:String=""varbody:someView{VStack{ImageClassifierView(modelURL:FruitClassifier.urlOfModelInThisBundle,
latestPrediction: $latestPrediction)Text(latestPrediction)}}}

Live Object Detection

Detect and locate objects in the live camera feed using a Create ML / Core ML Object Detector model.

1. Required Step: Drag in your Create ML / Core ML object detector model into Xcode.
2. Change MyObjectDetector below to the name of your Model.
3. detectedObjects holds the labels of the objects found in the latest frame; use it as you would any other state variable.

Utilise ObjectDetectorView

ObjectDetectorView(modelURL:MyObjectDetector.urlOfModelInThisBundle,
detectedObjects: $detectedObjects)

Full Example

import SwiftUI
import PrototypeKit
structObjectDetectorViewSample:View{@StatevardetectedObjects:[String]=[]varbody:someView{VStack{ObjectDetectorView(modelURL:MyObjectDetector.urlOfModelInThisBundle,
detectedObjects: $detectedObjects)ScrollView{ForEach(Array(detectedObjects.enumerated()), id: \.offset){ index, object inText(object)}}}}}

Need to know where each object is (for example, to draw bounding boxes)? Bind an array of DetectedObject instead of [String]. Each DetectedObject carries the label, a confidence (0–1), and a normalized boundingBox (CGRect, origin bottom-left as Vision reports it):

ObjectDetectorView(modelURL:MyObjectDetector.urlOfModelInThisBundle,
detectedObjects: $detectedObjects) // $detectedObjects is [DetectedObject]

Full Example (with bounding boxes)

import SwiftUI
import PrototypeKit
structObjectDetectorBoxesSample:View{@StatevardetectedObjects:[DetectedObject]=[]varbody:someView{ZStack{ObjectDetectorView(modelURL:MyObjectDetector.urlOfModelInThisBundle,
detectedObjects: $detectedObjects)GeometryReader{ geometry inForEach(Array(detectedObjects.enumerated()), id: \.offset){ _, object inletbox= object.boundingBox
Rectangle().stroke(.red, lineWidth:2)
// Vision's origin is bottom-left; SwiftUI's is top-left, so flip Y.
.frame(width: box.width * geometry.size.width,
height: box.height * geometry.size.height).position(x: box.midX * geometry.size.width,
y:(1- box.midY)* geometry.size.height).overlay(Text(object.label))}}}}}

Live Hand Pose Classification

Classify hand poses in real-time using a Create ML / Core ML hand action classifier.

1. Required Step: Drag in your Create ML / Core ML hand pose model into Xcode.
2. Change HandPoseClassifier below to the name of your Model.
3. You can use latestPrediction as you would any other state variable.

Utilise HandPoseClassifierView

HandPoseClassifierView(modelURL:HandPoseClassifier.urlOfModelInThisBundle,
latestPrediction: $latestPrediction)

Full Example

import SwiftUI
import PrototypeKit
structHandPoseClassifierViewSample:View{@StatevarlatestPrediction:String=""varbody:someView{VStack{HandPoseClassifierView(modelURL:HandPoseClassifier.urlOfModelInThisBundle,
latestPrediction: $latestPrediction)Text(latestPrediction)}}}

Live Action Classification

Classify a person's action from their body movement in real-time using a Create ML / Core ML Action Classifier model.

1. Required Step: Drag in your Create ML / Core ML action classifier model into Xcode.
2. Change ActionClassifier below to the name of your Model.
3. You can use latestPrediction as you would any other state variable.

An action unfolds over time, so ActionClassifierView detects body-pose keypoints with Vision and feeds a sliding window of frames (two seconds by default) into your model, updating latestPrediction as the window advances.

Utilise ActionClassifierView

ActionClassifierView(modelURL:ActionClassifier.urlOfModelInThisBundle,
latestPrediction: $latestPrediction)

If your model uses different feature names or a different prediction window, supply an ActionClassifierConfiguration:

ActionClassifierView(
modelURL:ActionClassifier.urlOfModelInThisBundle,
configuration:ActionClassifierConfiguration(predictionWindowSize:90),
latestPrediction: $latestPrediction
)

Full Example

import SwiftUI
import PrototypeKit
structActionClassifierViewSample:View{@StatevarlatestPrediction:String=""varbody:someView{VStack{ActionClassifierView(modelURL:ActionClassifier.urlOfModelInThisBundle,
latestPrediction: $latestPrediction)Text(latestPrediction)}}}

Live Text Recognition

Utilise LiveTextRecognizerView

LiveTextRecognizerView(detectedText: $detectedText)

Full Example

import SwiftUI
import PrototypeKit
structTextRecognizerView:View{@StatevardetectedText:[String]=[]varbody:someView{VStack{LiveTextRecognizerView(detectedText: $detectedText)ScrollView{ForEach(Array(detectedText.enumerated()), id: \.offset){ line, text inText(text)}}}}}

Live Barcode Recognition

Utilise LiveBarcodeRecognizerView

LiveBarcodeRecognizerView(detectedBarcodes: $detectedBarcodes)

Full Example

import SwiftUI
import PrototypeKit
structBarcodeRecognizerView:View{@StatevardetectedBarcodes:[String]=[]varbody:someView{VStack{LiveBarcodeRecognizerView(detectedBarcodes: $detectedBarcodes)ScrollView{ForEach(Array(detectedBarcodes.enumerated()), id: \.offset){ index, barcode inText(barcode)}}}}}

Live Animal Recognition

Recognize cats and dogs in real-time using the built-in Vision animal recognizer (no Core ML model required).

Utilise LiveAnimalRecognizerView

LiveAnimalRecognizerView(detectedAnimals: $detectedAnimals)

Full Example

import SwiftUI
import PrototypeKit
structAnimalRecognizerView:View{@StatevardetectedAnimals:[String]=[]varbody:someView{VStack{LiveAnimalRecognizerView(detectedAnimals: $detectedAnimals)ScrollView{ForEach(Array(detectedAnimals.enumerated()), id: \.offset){ index, animal inText(animal)}}}}}

Live Face Detection

Detect faces in real-time using the built-in Vision face detector (no Core ML model required).

Utilise LiveFaceDetectorView

LiveFaceDetectorView(faceCount: $faceCount)

Full Example

import SwiftUI
import PrototypeKit
structFaceDetectorView:View{@StatevarfaceCount:Int=0varbody:someView{VStack{LiveFaceDetectorView(faceCount: $faceCount)Text("Faces: \(faceCount)")}}}

Live Body Pose Detection

Detect human body poses in real-time using the built-in Vision human body pose request (no Core ML model required).

Utilise LiveBodyPoseDetectorView

LiveBodyPoseDetectorView(bodyCount: $bodyCount)

Full Example

import SwiftUI
import PrototypeKit
structBodyPoseDetectorView:View{@StatevarbodyCount:Int=0varbody:someView{VStack{LiveBodyPoseDetectorView(bodyCount: $bodyCount)Text("Bodies: \(bodyCount)")}}}

Live Rectangle Detection

Detect rectangular shapes (documents, cards, signs) in real-time using the built-in Vision rectangle detector (no Core ML model required).

Utilise LiveRectangleDetectorView

LiveRectangleDetectorView(rectangleCount: $rectangleCount)

Full Example

import SwiftUI
import PrototypeKit
structRectangleDetectorView:View{@StatevarrectangleCount:Int=0varbody:someView{VStack{LiveRectangleDetectorView(rectangleCount: $rectangleCount)Text("Rectangles: \(rectangleCount)")}}}

Live Sound Recognition

Required Step: Sound recognition uses the microphone, so you must add the Privacy - Microphone Usage Description (NSMicrophoneUsageDescription) key to your target's Info properties — follow the same steps as the camera setup above, using the microphone key instead. Without it, classification fails silently.

Utilise recognizeSounds modifier to detect sounds in real-time. This feature supports both the system sound classifier and custom Core ML models.

.recognizeSounds(recognizedSound: $recognizedSound)

For custom configuration, you can use the SoundAnalysisConfiguration:

.recognizeSounds(
recognizedSound: $recognizedSound,
configuration:SoundAnalysisConfiguration(
inferenceWindowSize:1.5, // Window size in seconds
overlapFactor:0.9, // Overlap between consecutive windows
mlModel: yourCustomModel // Optional custom Core ML model
))

Full Example

import SwiftUI
import PrototypeKit
structSoundRecognizerView:View{@StatevarrecognizedSound:String?varbody:someView{VStack{Text("Recognized Sound: \(recognizedSound ??"None")")}
// Attach the modifier to a view to start listening; updates `recognizedSound` live.
.recognizeSounds(recognizedSound: $recognizedSound)}}

Motion Tasks

Live Activity Classification

Classify the device's physical activity (for example walking, running, or standing still) in real-time from the accelerometer and gyroscope, using a Create ML / Core ML Activity Classifier.

1. Required Step: Drag in your Create ML / Core ML activity classifier model into Xcode.
2. Change ActivityClassifier below to the name of your Model.
3. You can use latestActivity as you would any other state variable.

Utilise the classifyActivity modifier to detect activity in real-time.

.classifyActivity(modelURL:ActivityClassifier.urlOfModelInThisBundle,
latestActivity: $latestActivity)

If your model's input/output feature names or sample rate differ from the Create ML defaults, supply an ActivityClassifierConfiguration:

.classifyActivity(
modelURL:ActivityClassifier.urlOfModelInThisBundle,
configuration:ActivityClassifierConfiguration(
sensorUpdateInterval:1.0/50.0, // Sensor sample rate in seconds (50 Hz)
predictionWindowSize:50 // Samples per prediction
),
latestActivity: $latestActivity
)

Note: Activity classification relies on CoreMotion and is available on iOS only. The modifier produces no visible content of its own — attach it to a view to drive classification and read latestActivity.

Full Example

import SwiftUI
import PrototypeKit
structActivityClassifierViewSample:View{@StatevarlatestActivity:String?varbody:someView{VStack{Text("Activity: \(latestActivity ??"Detecting…")")}
// Attach the modifier to a view to start classifying; updates `latestActivity` live.
.classifyActivity(modelURL:ActivityClassifier.urlOfModelInThisBundle,
latestActivity: $latestActivity)}}

Text Tasks

Analyse text on-device with Apple's Natural Language framework. Unlike the camera and sound features, these need no camera, microphone, permissions, or Core ML model — everything ships with the OS, and they work on both iOS and macOS. That makes them the gentlest way to get started with on-device ML.

Each is a View modifier that re-runs whenever the text you pass in changes, updating a binding with the result.

Sentiment Analysis

Score how positive or negative a piece of text is, from -1 (very negative) to 1 (very positive).

.analyzeSentiment(text: text, score: $score)

Full Example

import SwiftUI
import PrototypeKit
structSentimentView:View{@Statevartext:String="I love this!"@Statevarscore:Double=0varbody:someView{VStack{TextField("Type something", text: $text)Text("Sentiment: \(score, specifier:"%.2f")")}.analyzeSentiment(text: text, score: $score)}}

Language Identification

Detect the dominant language of a piece of text as a BCP-47 code (for example en, fr).

.identifyLanguage(text: text, language: $language)

Full Example

import SwiftUI
import PrototypeKit
structLanguageView:View{@Statevartext:String="Bonjour tout le monde"@Statevarlanguage:String?varbody:someView{VStack{TextField("Type something", text: $text)Text("Language: \(language ??"Detecting…")")}.identifyLanguage(text: text, language: $language)}}

Named Entity Recognition

Extract the people, places, and organizations mentioned in a piece of text.

.tagEntities(text: text, entities: $entities)

Full Example

import SwiftUI
import PrototypeKit
structEntitiesView:View{@Statevartext:String="Tim Cook announced the news in London."@Statevarentities:[String]=[]varbody:someView{VStack{TextField("Type something", text: $text)ScrollView{ForEach(Array(entities.enumerated()), id: \.offset){ index, entity inText(entity)}}}.tagEntities(text: text, entities: $entities)}}

Diagnostics & error handling 🩺

PrototypeKit is designed to fail gracefully — it will not crash your app on bad input:

• If a Core ML model can't be loaded (wrong URL, incompatible model), the affected view still shows the camera feed but produces no predictions. The failure is logged rather than fatal.
• Per-frame Vision and sound-classification errors are logged and skipped.
• If your app is missing the NSCameraUsageDescription key, the camera view shows an on-screen message explaining what to add instead of a black preview.

Diagnostics use Apple's unified logging system (os.Logger) under the subsystem com.prototypekit.PrototypeKit, so nothing is printed to your console in Release builds. To watch PrototypeKit's logs while developing:

log stream --predicate 'subsystem == "com.prototypekit.PrototypeKit"'

PrototypeKit only logs developer-facing diagnostics — never the contents of camera frames, audio, or recognized text.

FAQs

Is this production ready?

Not yet — PrototypeKit is intended for prototyping and idea validation, and it does not yet publish versioned releases. That said, it no longer crashes the host app on bad input (missing/invalid models, denied permissions, audio interruptions): those paths now degrade gracefully and log through os.Logger. See the CHANGELOG for details.