feat: expose scalabilityMode in VideoPublishOptions - #1122
davibittencourtome wants to merge 1 commit into
Conversation
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
308d540 to
15c1b0a
Compare
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
6aaea2c to
1ccb46d
Compare
computeVideoEncodings hardcodes L3T3_KEY for camera whenever the preferred codec is SVC, so an application has no way to choose the mode. livekit-client exposes exactly this knob as VideoPublishOptions.scalabilityMode. It matters against SFUs that turn off the dependency descriptor for a track published with L3T3_KEY: every subscriber of that publisher then freezes until the track is republished, and the publisher sees nothing wrong on its side. Publishing a single spatial layer (L1T3) avoids the whole situation, and is already what the JS client allows. Add VideoPublishOptions.scalabilityMode — nil keeps the current behavior — and honor it in the SVC branch of computeVideoEncodings, including the overrideVideoCodec path used by backup codecs and republish. Screen share keeps forcing L1T3 regardless of the option: WebRTC does not publish SVC screen share with multiple spatial layers, so letting a shared VideoPublishOptions override it would emit no frames at all. The initializer that existed before this property is preserved as a convenience init with its original Objective-C selector, forwarding scalabilityMode: nil, so binaries compiled against the previous framework keep resolving it — same pattern as AudioCaptureOptions. ScalabilityMode gains Sendable, required by the property on a Sendable-conforming class.
1ccb46d to
55f7d06
Compare
| /// configuration `livekit-client` (JS) exposes as `scalabilityMode: 'L1T3'`. Motivation: | ||
| /// with `L3T3_KEY` some SFU versions turn the dependency descriptor off for the track | ||
| /// and subscribers freeze; a single spatial layer avoids it. | ||
| public let scalabilityMode: ScalabilityMode? |
There was a problem hiding this comment.
ScalabilityMode? (an Optional of an @objc enum) is not Objective-C-representable, so @objcMembers silently drops both this property and the new designated initializer from the generated header — verified with -emit-objc-header: @interface VideoPublishOptions has no scalabilityMode and exactly one init, the legacy 10-arg one — which also means the convenience init below buys nothing, and modelling this as a non-optional enum with an .auto case (the pattern the sibling degradationPreference already uses) would fix both at once.
|
|
||
| /// Scalability mode used when publishing the **camera** with an SVC codec (VP9/AV1). | ||
| /// `nil` keeps the default, `L3T3_KEY`. Ignored for non-SVC codecs (VP8/H264), which use | ||
| /// simulcast instead, and ignored for screen share, which always uses `L1T3` because |
There was a problem hiding this comment.
This invariant can't hold on the backup-codec path: LocalParticipant.publish(additionalVideoCodec:) (LocalParticipant.swift:520) calls computeVideoEncodings without isScreenShare:, so a screen-share track with a preferredBackupCodec gets its backup stream computed as camera — camera encoding, camera simulcast layers and camera presets, not just the scalability mode — unlike line 648, which correctly passes isScreenShare: track.source == .screenShareVideo.
| // there — a multi-spatial mode publishes no frames at all. An explicit | ||
| // `publishOptions.scalabilityMode` only applies to camera encodings (parity with | ||
| // livekit-client `scalabilityMode`). | ||
| let scalabilityMode: ScalabilityMode = isScreenShare ? .L1T3 : (publishOptions.scalabilityMode ?? .L3T3_KEY) |
There was a problem hiding this comment.
Since videoCodec is overrideVideoCodec ?? publishOptions.preferredCodec (line 30), an app that sets scalabilityMode but leaves preferredCodec at its nil default never reaches this branch even when SDP negotiates VP9 — the option is a complete no-op, and the only log here is .debug and prints the resolved mode rather than the fact that the caller's value was dropped, so a .warning when publishOptions.scalabilityMode is non-nil and discarded would save someone a long debugging session.
Problem
Utils.computeVideoEncodingshardcodes the scalability mode whenever the preferred codec is SVC:ScalabilityModeis public andRTC.createRtpEncodingParametersalready takes it, but no public option feeds it, so a camera publish is alwaysL3T3_KEY.livekit-clientexposes the same knob asVideoPublishOptions.scalabilityMode, so an app can be configured on the web and not on iOS.This is not theoretical for us. Against livekit-server 1.9.12 we see the SFU turn the dependency descriptor off for tracks published with
L3T3_KEYand never turn it back on; every subscriber of that publisher then freezes, while the publisher's own preview looks fine and its outbound stats stay healthy. Our web clients work around it by publishingL1T3(one spatial layer, temporal scalability kept). The iOS apps could not, and the only remaining lever was falling back to VP8, which costs noticeably more bitrate for the same quality.Fix
Add
VideoPublishOptions.scalabilityMode(defaults tonil, which keeps the current behavior exactly) and honor it in the SVC branch:The option is threaded through the
overrideVideoCodecpath as well, so it still applies when the backup codec or a republish recomputes the encodings — otherwise the mode would silently revert on the very paths where the freeze shows up.ScalabilityModegainsSendable.VideoPublishOptionsis aSendable-conforming final class, so a stored property of a non-Sendableenum does not compile; the enum is a plainInt-backed value, so this is not a behavior change.The property is included in
isEqualandhash, matching the other publish options.Testing
New
VideoEncodingsScalabilityModeTests(swift-testing):svcDefaultsToL3T3KeyForCamera/svcDefaultsToL1T3ForScreenShare— unchanged defaults when the option isnil.explicitScalabilityModeWinsForSvcCamera—L1T3reaches the encoding parameters.explicitScalabilityModeSurvivesCodecOverride— the option still applies on theoverrideVideoCodecpath.scalabilityModeIsIgnoredForNonSvcCodec— VP8 still produces simulcast layers, none carrying a scalability mode.optionsEqualityIncludesScalabilityMode— equality and hash account for the new property.Red/green checked: reverting only the
Utils+VideoEncodings.swiftchange makes the two explicit-mode tests fail and leaves the four others passing. Run withswift test --filter VideoEncodingsScalabilityModeTestson macOS.I did not find an existing issue for this; happy to open one if you prefer to track it separately.