Video Music

A modern, evolved clone of the Atari Video Music (C-240, 1977) — a dedicated audio visualizer that generates real-time graphics driven by music. Built as a native tvOS application for Apple TV 4K.

The app plays music from the user's Apple Music library and renders generative visuals on the TV in response to the audio signal. Six distinct visual modes can be cycled via the Siri Remote.

---

Architecture

Three systems. That's it.

1. Music Playback — MusicKit

• Uses ApplicationMusicPlayer for independent playback (doesn't affect the system Music app)
• Requests Apple Music authorization on launch
• Browse albums, playlists, and search the Apple Music catalog
• Now Playing overlay with track title, artist, album art — fades out after 4 seconds
• Standard transport controls via Siri Remote

2. Audio Analysis — AVAudioEngine + FFT

Real-time frequency and amplitude extraction using AVAudioEngine with vDSP (Accelerate framework).

Extracted data every frame:

Parameter	Range	Description
Bass	0.0–1.0	20–250 Hz (kick drums, sub-bass)
Low-Mid	0.0–1.0	250–500 Hz (warmth, body)
Mid	0.0–1.0	500–2000 Hz (vocals, snare)
High-Mid	0.0–1.0	2000–6000 Hz (presence, articulation)
High	0.0–1.0	6000–20000 Hz (air, cymbals)
Amplitude	0.0–1.0	Overall RMS level
Transient	0.0–1.0	Sharp attack impulse (decays quickly)
Spectral Centroid	0.0–1.0	Weighted avg frequency (dark↔bright)

All values smoothed with exponential moving average to prevent visual jitter.

Audio Tap Approach

MusicKit's ApplicationMusicPlayer does not expose PCM audio buffers. Apple provides no documented API to tap into MusicKit's audio output stream.

Approach used: Microphone Loopback

• AVAudioEngine taps the device's input node (microphone)
• The Apple TV 4K's built-in Siri microphone captures the room audio
• Music playing through speakers/HDMI is picked up and analyzed

Limitations:

• Requires Apple TV 4K (has built-in mic for Siri)
• Audio quality depends on room acoustics
• Background noise can affect analysis
• Slight latency (~50ms) between playback and visual response

Alternative approaches investigated:

1. AVAudioEngine tap on outputNode — crashes because MusicKit owns the audio session
2. MTAudioProcessingTap — requires AVPlayer/AVPlayerItem, not compatible with MusicKit
3. AVAudioPlayerNode with MusicKit stream — MusicKit doesn't expose a stream URL

Fallback: When no microphone is available (e.g., Simulator), the app runs in simulated mode with procedurally generated audio data that produces plausible visualization behavior.

3. Visual Rendering — Metal

Full-screen Metal rendering at native resolution/refresh rate.

• MTKView as the primary rendering surface
• Fragment shader per visual mode (fullscreen quad approach)
• Reads AudioAnalysisData every frame via shared Uniforms struct
• ~1.5 second crossfade transition when switching modes
• Zero UI chrome during visualization — full bleed edge to edge

---

Visual Modes

#	Name	Character
0	C-240	Lissajous curves, CRT-era colors, analog warmth. Homage to the original.
1	Membrane	Organic undulating surface, bass deformation, spectral color shifts
2	Particle Field	Thousands of particles, transient bursts, gravity wells, fading trails
3	Tunnel	Raymarched wormhole, amplitude-driven velocity, beat-synced color cycling
4	Glitch Grid	Hard-edged geometric grid, transient displacement, color inversion
5	Drift	Slow-evolving noise nebula, minimal audio reactivity, ambient background

Every mode includes a drift parameter — slow autonomous evolution that prevents static-looking output even when audio is steady.

---

Siri Remote Controls

Input	Action
Play/Pause button	Play/pause music
Left on clickpad	Previous track
Right on clickpad	Next track
Up on clickpad	Next visual mode
Down on clickpad	Previous visual mode
Menu button	Show/hide music browser
Select (center click)	Show Now Playing info briefly

---

Screensaver Behavior

tvOS does not expose a public API for third-party screensaver registration. Instead, the app implements auto-dim simulation:

• After 30 seconds of no remote input, all UI is hidden
• The visualizer runs full-screen without any overlays
• If music is playing, it visualizes the music
• If no music is playing, the current mode runs with drift-only motion (ambient behavior)
• Any remote interaction immediately exits screensaver mode

---

Project Structure

VideoMusic/
├── VideoMusic.xcodeproj
├── VideoMusic.entitlements # MusicKit entitlement
├── VideoMusic/
│ ├── VideoMusic-Bridging-Header.h # Exposes ShaderTypes.h to Swift
│ ├── App/
│ │ ├── VideoMusicApp.swift # @main app entry point
│ │ └── ContentView.swift # Root view — orchestrates all systems
│ ├── Music/
│ │ ├── MusicManager.swift # MusicKit auth, library, playback
│ │ ├── MusicBrowserView.swift # Album/playlist/search browser UI
│ │ └── NowPlayingView.swift # Now Playing overlay (auto-fades)
│ ├── Audio/
│ │ ├── AudioEngine.swift # AVAudioEngine setup + mic loopback
│ │ ├── AudioAnalysisData.swift # Shared audio data (thread-safe)
│ │ └── FFTProcessor.swift # vDSP FFT + band extraction
│ ├── Visual/
│ │ ├── VisualRenderer.swift # MTKView + Metal pipeline
│ │ ├── ShaderTypes.h # Shared Swift/Metal data types
│ │ ├── ModeTransitionManager.swift # Crossfade between modes
│ │ └── Shaders/
│ │ ├── C240Mode.metal # Mode 0 + shared vertex shader
│ │ ├── MembraneMode.metal # Mode 1
│ │ ├── ParticleMode.metal # Mode 2
│ │ ├── TunnelMode.metal # Mode 3
│ │ ├── GlitchGridMode.metal # Mode 4
│ │ └── DriftMode.metal # Mode 5
│ ├── Remote/
│ │ └── RemoteInputHandler.swift # Siri Remote via GameController
│ └── Resources/
│ ├── Assets.xcassets
│ └── Info.plist
└── README.md

---

Tech Stack

Component	Technology
Language	Swift 5, Metal Shading Language
UI	SwiftUI
Music	MusicKit (ApplicationMusicPlayer)
Audio Analysis	AVFoundation, AVFAudio, Accelerate (vDSP)
Rendering	Metal, MetalKit
Input	GameController framework
Target	tvOS 17.0+
Device	Apple TV 4K (all generations)

---

Setup

1. Open VideoMusic.xcodeproj in Xcode 15+
2. In the project's Signing & Capabilities:
   • Set your Development Team
   • The MusicKit capability should already be configured
3. In the Apple Developer Portal:
   • Register the App ID com.fladrycreative.videomusic
   • Enable the MusicKit App Service
4. Build and run on Apple TV Simulator or a real Apple TV 4K
5. On first launch, approve the Apple Music access prompt

Note: The Simulator will run in simulated audio mode (no microphone). For real audio-reactive behavior, deploy to a physical Apple TV 4K.

---

For Shader Artists

The shaders are the soul of this app and will be iterated heavily. Here's what you need to know:

Data Flow

Audio → FFTProcessor → AudioAnalysisData → Uniforms (CPU) → Fragment Shader (GPU)

Every fragment shader receives a Uniforms struct containing:

• All 5 frequency bands + amplitude + transient + spectral centroid
• Time, delta time, resolution
• Current mode, transition progress
• Drift parameter (slow autonomous evolution)

Editing Shaders

1. Each mode is a self-contained .metal file
2. The shared vertex shader lives in C240Mode.metal
3. ShaderTypes.h defines the Uniforms struct — shared between Swift and Metal
4. Fragment function naming convention: fragment[ModeName] (e.g., fragmentC240)
5. All modes receive the same Uniforms data — the difference is how they interpret it

Adding a New Mode

1. Create NewMode.metal in Visual/Shaders/
2. Add the fragment function (follow existing patterns)
3. Add the fragment name to the fragmentNames array in VisualRenderer.swift
4. Increment modeCount in ModeTransitionManager.swift
5. Add the mode name to modeNames array
6. Add the enum case in ShaderTypes.h

---

What's NOT Included (Coming Later)

• Companion iOS/watchOS app
• OSC / network / external control
• Settings screen or preferences
• Social features, sharing, or recording
• In-app purchases or monetization

---

License

Copyright 2026 Fladry Creative. All rights reserved.