Skip to content

Data tracks UniFFI - #1034

Merged
ladvoc merged 70 commits into
mainfrom
ladvoc/data-tracks-uniffi
Jul 10, 2026
Merged

ladvoc merged 70 commits into
mainfrom
ladvoc/data-tracks-uniffi

Conversation

@ladvoc

@ladvoc ladvoc commented Apr 22, 2026

Copy link
Copy Markdown
Contributor

Summary of changes:

  • Exposes data tracks core functionality through livekit-uniffi
    • This will eventually enable the following clients to share the Rust implementation: Swift, Kotlin, React Native, Flutter
  • Minor changes to the data tracks crate to support this

Resolves CLT-2472

@ladvoc

ladvoc commented Jun 30, 2026

Copy link
Copy Markdown
Contributor Author

Join-time tracks. No join handler on RemoteDataTrackManager, even though the reference SDK runs event_from_join on connect (rtc_session.rs:531) — so a foreign client joining a room with existing data tracks won't see them. Either add a handle_sfu_join, or just feed join.other_participants through the existing handle_sfu_participant_update (what JS effectively does). Identical under the hood — both hit event_from_participant_info. Preference?

Rust has these additional helpers to keep the interface between the rest of the client SDK and the data tracks managers as uniform as possible; instead of caring about specific fields in each response type, you hand over the whole response, and the logic of how the required information is extracted is kept as an implementation detail. A few other advantages:

  • If the data track managers ever need additional information not extracted from participant info, no update to this interface will be needed.
  • Local participant identity isn't needed when handling a join response but it is when handling a participant update; this gets modeled in the signature of each method.

I implemented handle_sfu_join_response in 8abcf90. While these helpers don't add any overhead in the Rust client integration since they accept the responses by reference, in an FFI context, there is additional overhead because the whole response must be serialized rather than just the relevant fields—I'm open to changing this if the additional overhead is a concern.

@ladvoc

ladvoc commented Jun 30, 2026

Copy link
Copy Markdown
Contributor Author

Caveats: no released livekit-server (incl. 1.13.2) fills ParticipantInfo.data_tracks in the join, so join discovery isn't verifiable e2e atm — is the SFU side still unreleased?

Data tracks is fully released in OSS and Cloud now so this would be unexpected. Do you have any repro steps for this?

@pblazej

pblazej commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

Do you have any repro steps for this?

No repro — false alarm on my side. The join does carry data_tracks on 1.13.2; my test was discarding the returned LocalDataTrack (_ = try await publishDataTrack(...)), which unpublishes it on drop, so the track was already gone before the late joiner arrived. Holding it, join-time discovery works. Both handle_sfu_join_response and publish_responses_for_sync_state are wired up and green e2e now.

That RAII-on-drop is really the last open question: should FFI consumers mirror Rust's semantics (drop = unpublish) or go "publication-like" (JS keeps the publication in its manager until an explicit unpublish)? We went publication-like on the Swift side so callers aren't forced to retain the handle — this diff shows it: livekit/client-sdk-swift@3e3043e

Framing it as "what should the convention be" rather than "should Rust change": the core's RAII is correct/idiomatic for Rust consumers — the divergence is about what the FFI bindings present. JS and (now) Swift both land on publication-like, so there's a de-facto convention worth making explicit.

For concreteness, the Swift lifecycle surface:

API Semantics
publishDataTrack(name:) publication-like — SDK retains it; discarding the handle does not unpublish
track.unpublish() / await track.waitForUnpublish() explicit unpublish + await
withDataTrack(name:) { … } scoped RAII — auto-unpublishes when the block exits

withDataTrack is the compromise: it keeps Rust's RAII where it's genuinely a feature (an explicit lexical scope where auto-cleanup is exactly what you want), while the default publishDataTrack stays publication-like so discarding the handle doesn't surprise-unpublish.

@pblazej
pblazej marked this pull request as ready for review July 1, 2026 08:06

@pblazej pblazej left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see any functional gaps here preventing Swift integration

The rest is mostly consumer-side discussion (like the RAII pattern mentioned above).

Before merging please update both SPM_SIZE_LIMIT_BYTES and ANDROID_SIZE_LIMIT_BYTES with the final sizes.

Approximate sizes @ the latest commit:

  • SPM_SIZE_LIMIT_BYTES = 1152136
  • ANDROID_SIZE_LIMIT_BYTES = 1250710

@pblazej
pblazej requested a review from hiroshihorie July 7, 2026 13:46
@pblazej

pblazej commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

@hiroshihorie cc what's your take on applying that in flutter (as the infra is ~complete)?

devin-ai-integration[bot]

This comment was marked as resolved.

ladvoc and others added 5 commits July 9, 2026 14:33
Add subscribe_with_options on RemoteDataTrack and a DataTrackSubscribeOptions
record (buffer_size) so foreign consumers can tune the internal frame buffer,
matching the core and JS subscribe APIs.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@ladvoc
ladvoc force-pushed the ladvoc/data-tracks-uniffi branch from d0c8d7b to eda4a11 Compare July 9, 2026 21:45
devin-ai-integration[bot]

This comment was marked as resolved.

@ladvoc
ladvoc force-pushed the ladvoc/data-tracks-uniffi branch from 163cdd9 to 61bd528 Compare July 10, 2026 17:40
@ladvoc

ladvoc commented Jul 10, 2026

Copy link
Copy Markdown
Contributor Author

I have temporarily disable Dart tests in CI since they are currently failing. In the interest of getting this merged, we can address in a follow-up PR.

@ladvoc
ladvoc merged commit 5067671 into main Jul 10, 2026
29 checks passed
@ladvoc
ladvoc deleted the ladvoc/data-tracks-uniffi branch July 10, 2026 18:26
pblazej added a commit that referenced this pull request Jul 30, 2026
…1300)

- build-android-*: cargo-make's `extend` replaces the parent env map
rather than merging it, so `env = { TARGET = ... }` silently dropped
ANDROID_RELEASE_FLAG and RUSTFLAGS — `--profile release` produced debug
.so files and android-copy-jniLibs found nothing in target/*/release.
Derive the flag from CARGO_MAKE_PROFILE in the script instead.
- android-bindgen-kotlin: TARGET leaks across cargo-make tasks, so
bindgen-kotlin's `build` dependency cross-compiled for Android with the
host linker ("cannot find -llog"). Pin TARGET back to the host triple
and keep symbols, which library-mode bindgen needs on Linux.
- Raise both size gates to measured values: ios-arm64 is 1088 KiB and
arm64-v8a 1174 KiB, having grown past the limits set in #1171 when the
data-track UniFFI surface landed in #1034.
- Check out inputs.tag_name in both reusable workflows;
workflow_dispatch was building the dispatch ref (main) rather than the
requested tag.

### Before you submit your PR

Make sure the following is true before submitting your PR:

- [ ] I have read the [contributing
guidelines](https://github.com/livekit/rust-sdks/blob/main/CONTRIBUTING.md)
and validated that this PR will be accepted.
- [ ] I have read and followed the principles regarding breaking
changes, testing, and code quality.

### PR description

Describe the changes in this PR. Explain what the PR is meant to solve
and how to reproduce the issue in the first place.

### Breaking changes

If this PR introduces breaking changes, list them here and document the
rationale for introducing such a change.

### MSRV

If the PR modifies the crate's MSRV (Minimum Supported Rust Version),
document it here.

### Testing

Ideally, unit test the code you add, but ensure you're not repeating
existing test cases. Use as many already written scaffolding, utilities
as possible; write your own, when needed. If external services, APIs,
tokens are required (e.g., running an LK server instance), provide the
necessary information. Make sure your tests perform useful,
context-aware assertions and do not simply emulate "happy paths".

### Async

We want the project to be runtime-agnostic, so please reuse what's
already in
[livekit-runtime](https://github.com/livekit/rust-sdks/blob/main/livekit-runtime/)
and feel free to add anything missing. It's ok to use Tokio directly,
when writing unit tests, if necessary. When testing, do not use
artificial delays for the state to "catch up"; instead, respect the
event flow and subscribe properly using channels or other mechanisms.

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
pblazej added a commit to livekit/client-sdk-swift that referenced this pull request Aug 17, 2026
Adds [data tracks](livekit/rust-sdks#1034) to
the Swift SDK, implemented on top of the Rust data-track engine via
UniFFI ([livekit-uniffi-xcframework
0.1.8](https://github.com/livekit/livekit-uniffi-xcframework/releases/tag/0.1.8)).

## API

```swift
// Publish + push
let track = try await room.localParticipant.publishDataTrack(name: "telemetry")
try track.tryPush(frame: DataTrackFrame(payload: data))          // non-blocking
try await track.send(contentsOf: frames)                          // AsyncSequence, drop-on-full by default

// Optionally declare what the frames are, and make the schema resolvable
let schema = DataTrackSchemaId(name: "telemetry.v1", encoding: .protobuf)
try await room.localParticipant.defineSchema(schema, definition: protoSource)
let typed = try await room.localParticipant.publishDataTrack(
    name: "typed",
    options: DataTrackPublishOptions(schema: schema, frameEncoding: .protobuf)
)

// Subscribe + receive
let stream = try await remoteTrack.subscribe(bufferSize: 16)
for await frame in stream { ... }                                 // DataTrackStream is an AsyncSequence

// A subscriber can resolve the schema its publisher declared
if let declared = remoteTrack.info.schema {
    let definition = try await room.localParticipant.getSchema(declared, publishedBy: remoteTrack.publisherIdentity)
}
```

Remote publish/unpublish surface on `RoomDelegate` and
`ParticipantDelegate`; subscribed tracks are on
`RemoteParticipant.dataTracks`, keyed by track name. Everything is
Objective-C compatible (`DataTrackObjCTests` covers the surface).

## Design decisions

**Rust owns the protocol, Swift owns the session.** The UniFFI managers
(`LocalDataTrackManager`/`RemoteDataTrackManager`) implement the
publish/subscribe state machine, packetization, and E2EE. The SDK
contributes what only it has: the signal connection, the WebRTC data
channels, participant identity, and delegate fan-out. Concretely, a
session-scoped `DataTracks` coordinator (one per `Room`,
`Room+DataTrack.swift`) feeds SFU signal messages into the managers,
forwards their outbound requests through `SignalClient` (serialized, so
publish/unpublish ordering is preserved), and pumps packets between the
managers and the DTP data channel.

**Thin public wrappers over the bindings.** The generated bindings are
`internal import`ed and every public type (`LocalDataTrack`,
`RemoteDataTrack`, `DataTrackFrame`, `DataTrackStream`, `DataTrackInfo`,
schema/encoding types, error enums) is a small SDK-owned wrapper (~900
lines total). Tradeoff considered: exposing the generated types directly
would save the layer, but pins the public API to regenerated code we
don't control, and UniFFI emits Swift-only structs — `NSObject` wrappers
are what makes the API reach Objective-C. The wrappers also carry the
SDK idioms (lowerCamel error enums, `Track.Sid`-style identifiers,
delegate naming, value semantics).

**Publication lifetime follows the handle (RAII).** The returned
`LocalDataTrack` *is* the publication: releasing the last reference
unpublishes it, as does calling `unpublish()`. This matches rust-sdks,
where the handle drops the publication. Consequently there are no local
publish/unpublish delegate events — Rust dispatches room events only for
*remote* data tracks, and the handle (`isPublished`,
`waitForUnpublish()`) is the local observer. `withDataTrack(name:body:)`
scopes a publication to a block for the common case.

**Retain-cycle handling at the FFI boundary.** UniFFI callback
interfaces hold their Swift delegate strongly (no weak references over
FFI). A dedicated `ManagerDelegate` object references the `Room` and
coordinator weakly, making it the designated weak link so `manager →
delegate → room → manager` cycles can't leak (`Room+DataTrack.swift`
documents the shape).

**Reconnect semantics.** Quick reconnect preserves publications via
`SyncState.publishDataTracks` (the managers replay their publish
responses); full reconnect republishes and re-asserts subscriptions. The
`DataTracks` subsystem survives reconnects (channels are swapped in) and
is torn down only on real disconnect, which also unpublishes remote
tracks and notifies delegates. The publisher-channel readiness gate is
re-armed — not failed — on teardown, so a `publishDataTrack` issued
mid-reconnect waits for the new channel instead of racing a dead
transport. A local full reconnect recreates participants but *not* their
data tracks: those are detached silently and re-attached, so no spurious
unpublish is reported.

When a *remote publisher* full-reconnects, the Rust manager treats the
republication as a SID reassignment: the subscriber's existing
`RemoteDataTrack` survives with its SID rewritten in place and active
subscriptions transparently re-requested — no unpublish/republish events
fire. This is why `RemoteParticipant.dataTracks` is keyed by **name**,
not SID (Rust documents SIDs as unstable across reconnects;
client-sdk-js keys by name for the same reason).

**Backpressure.** `tryPush` is non-blocking and throws `queueFull`,
handing the rejected frame back so it can be retried (parity with Rust's
`PushFrameError::into_frame`). `send(contentsOf:)` defaults to dropping
frames when the queue is full (DTP is lossy by design — unbounded
buffering would trade a dropped frame for unbounded latency). On the
channel, `DataTrackFrameSender` meters packets out on buffered-amount
events against an 8 KiB low-water mark, keeping send latency bounded
while letting a frame of any size stream out: at most one frame waits
while another drains, a newer frame evicts the waiting one
(drop-oldest), and frames are handled whole so a partial frame is never
left on the wire. This mirrors `DataChannelSender` in rust-sdks;
client-sdk-js instead blocks the producer, which isn't available here
(the producer is a fire-and-forget FFI callback with no backpressure
channel). `DataTrackFrameSenderTests` pins these semantics, including
the cross-SDK divergence. Subscribe-side buffer is caller-configurable
(`subscribe(bufferSize:)`).

**E2EE.** When the room has E2EE configured, the same key provider
drives data-track frame encryption via UniFFI
`EncryptionProvider`/`DecryptionProvider` shims;
`DataTrackInfo.usesE2ee` reflects it. Unlike data-channel payloads (a
per-message property), data-track encryption is a track-level protocol
property subscribers key their decryption on, so it can't be consulted
per frame: the publishing manager is built on the *first publish* and
captures the toggle then, which is late enough for a
`setE2EEEnabled(true)` issued after connecting to apply. Toggling
between publishes does not, and `Room.setE2EEEnabled` documents that.
The cryptor resolves `Room.e2eeManager` on each call rather than
capturing it at connect, so a manager assigned late is picked up for
both encryption and decryption, and a missing one throws instead of
silently sending plaintext.

## Protocol update

The `protocol` submodule moves v1.45.8 → v1.50.4 (regenerated with `make
proto`). Two reasons:

- `DataTrackInfo` gained `frame_encoding` and `schema` after v1.45.8.
Because `handleParticipantUpdate` re-serializes the parsed
`ParticipantInfo` to hand the managers raw bytes, the older protocol
silently stripped both fields before Rust saw them — declared metadata
never reached subscribers. (The SFU was echoing it correctly; this was
ours.)
- It brings `DataBlob`/`StoreDataBlobRequest`/`GetDataBlobRequest`,
which `defineSchema`/`getSchema` are built on.

The general hazard remains and is worth a follow-up: any
`ParticipantInfo` field newer than the pinned protocol is lost on the
way to the managers. Threading the received wire bytes instead of
re-serializing removes both the loss and a redundant encode.

## Size impact

App Size job measured **16.28 MB** for the SDK's uncompressed `.app`
delta before the protocol update; the budget moves 16 → 16.5 MB to leave
room for it. That figure includes the nanopb protocol migration from
`main`, which cut ~1.8 MB; the data-track share is the 0.0.6 → 0.1.8
UniFFI bump plus this integration (~+0.4 MB: compiled bindings at the
dead-strip floor plus the Rust dylib growth).

Isolating data tracks behind a Swift package trait was evaluated and
rejected for now: the core SDK already depends on LiveKitUniFFI (tokens,
log forwarding), both UniFFI components live in one dylib, and the
bindings are one module — a trait could only gate the wrapper layer that
dead-strip already removes. Revisit if the Rust side ships a
feature-split artifact.

## Known upstream issue

`LocalDataTrack.tryPush` cannot surface the FFI's own error:
`PushFrameErrorReason` is declared in the `livekit_datatrack` UniFFI
component while `try_push` lives in `livekit_uniffi`, and the
cross-component error lift fails, throwing
`UniffiInternalError.bufferOverflow` instead. Left unhandled this leaked
an FFI-internal type through the public API and broke
`send(contentsOf:)`'s drop-on-full policy, which inspects the reason.
The wrapper recovers the applicable reason from `isPublished`; the typed
catch remains first and takes over once the bindings are fixed. Worth
filing against livekit-uniffi.

## Not in this PR

- `tryPush` hands the rejected frame back, but the FFI still drops
Rust's own `PushFrameError` payload.
- Remote pipeline options (`max_partial_frames`) aren't exported by
UniFFI yet.
- Follow-ups filed from review: sharing the buffering state machine with
`DataChannelPair`, and threading wire bytes through the signal path
instead of re-serializing parsed join/participant updates.

## Testing

Two unit suites run without a server: `DataTrackFrameSenderTests` covers
the outbound drain, and `DataTrackSendTests` covers
`send(contentsOf:)`'s queue-full policy against a sink that rejects on
demand — deliberately not by saturating a live pipeline, which made the
outcome depend on how fast the SFU drained.

End-to-end tests against `livekit-server --dev` across six suites:
publish/subscribe/roundtrip (small + multi-packet frames), publish
options and schema/encoding metadata round-tripping through the SFU,
schema definition storage and resolution, error cases (duplicate name,
unauthorized, disconnected, push after unpublish, saturated queue), E2EE
on, off, and toggled after connect, delegate events, unpublish paths,
RAII handle release, `withDataTrack` scoping including cancellation,
subscribe buffer clamping and drop-oldest eviction, concurrent
multi-track push and 64-track publish, plus lifecycle: pre-join
publications, publish during a full reconnect, publisher full-reconnect
SID reassignment, quick-reconnect sync state, remote tracks surviving
the local client's full reconnect, and room moves. `DataTrackObjCTests`
covers the Objective-C surface including NSError bridging and value
semantics.

`DataTrackStream` only ends when its track is unpublished, so every read
goes through bounded `next(within:)`/`collect(_:)` helpers — a lost
frame on an unreliable channel should fail a test, not hang the job. The
repeated two-room publish/subscribe preamble lives in a
`withPublishedDataTrack` fixture; tests that control *when* the publish
happens keep driving `withRooms` directly, since that timing is what
they exercise.

Schema definitions are stored as participant data blobs, which the
server keeps behind `enable_participant_data_blob` — enabled for CI, and
documented in `AGENTS.md` for local runs.

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
hiroshihorie added a commit that referenced this pull request Sep 8, 2026
### Before you submit your PR

Make sure the following is true before submitting your PR:

- [x] I have read the [contributing
guidelines](https://github.com/livekit/rust-sdks/blob/main/CONTRIBUTING.md)
and validated that this PR will be accepted.
- [x] I have read and followed the principles regarding breaking
changes, testing, and code quality.

### PR description

Automates the Dart/Flutter half of the livekit-uniffi release
(CLT-2872), the release-side prerequisite for
livekit/client-sdk-flutter#1160.

- **Make the generated package publishable.** Release builds omit the
dev dylib and `publish_to`, gain LICENSE/README/CHANGELOG and real
version constraints, and fail closed for unknown profiles. A
`dart-clean` task keeps stale host dylibs out of release packages, and a
shipped `analysis_options.yaml` silences lints that only fire on
generated code.
- **Re-enable the Dart tests** disabled since #1034. The uniffi-dart pin
on main already carries the multi-crate codegen fixes from upstream, so
this PR no longer changes the pin.
- **Add `uniffi-dart-publish.yml`**, a tag-push workflow (pub.dev only
accepts OIDC tokens from tag-push runs). It resolves and validates the
tag against the crate version, builds every `build-<triple>.zip` through
the reusable `uniffi-cdylib.yml` and attaches them to the release, then
builds the Dart package from a copy outside the work tree (`packages/`
is gitignored and pub archives from git's file listing), validates it,
and publishes. The publish job depends on the asset job, so there is
nothing to poll and a package can never publish without its assets.
- **Harden `uniffi-cdylib.yml`.** It checks out the release tag rather
than the dispatch ref, builds the Linux libraries inside manylinux_2_28
containers (glibc 2.28 floor, where ubuntu-latest gives 2.39 and breaks
Debian 12 / Ubuntu 22.04 / AL2023), builds aarch64 Linux on a native
arm64 runner, and installs cargo-ndk as a prebuilt binary.
- **Pin the Dart SDK** to 3.13.1 in both the test and publish workflows
so PR CI validates the SDK the publish runs with.

**Publishing stays disabled** (`PUBLISH_ENABLED=false`, every run stops
at `--dry-run`) until the first manual publish creates the package on
pub.dev and automated publishing is configured. Runbook in the workflow
header.

### Breaking changes

None. Dev-profile `cargo make dart-package` behaves as before.

### MSRV

No changes.

### Testing

- Dev build: all 5 Dart FFI tests pass, same flow now re-enabled in CI.
- Release build: `dart pub publish --dry-run` exits 0 with zero warnings
and no dylib in the package.
- Full release matrix dry run from this branch against the v0.1.9 tag:
all 12 cdylib targets plus the Swift and Android jobs passed with
`dry_run=true`
(https://github.com/livekit/rust-sdks/actions/runs/34203702564). The
CI-built Linux libraries were downloaded and verified: bare `.so` at the
zip root, sha256 sidecars match, highest required symbol version
GLIBC_2.28 on both architectures.
- Consumer: livekit/client-sdk-flutter#1160's uniffi tests pass against
the generated package.
- Not exercisable before merge: the publish workflow's own job wiring,
because GitHub only registers dispatchable workflows from the default
branch. Plan is to dispatch it from main with `dry_run=true` before the
next livekit-uniffi release. Whether knope's API-created tag fires the
push trigger is confirmed only by the first real release. Both cost
nothing while publishing is disabled.

### Async

No async code added; the crate surface is untouched.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants