Skip to content

Add the MLX backend product to the SwiftPM package template - #22259

Closed
shoumikhin wants to merge 4 commits into
swiftpmfrom
swiftpm-mlx-metallib-fix-v2
Closed

Add the MLX backend product to the SwiftPM package template#22259
shoumikhin wants to merge 4 commits into
swiftpmfrom
swiftpm-mlx-metallib-fix-v2

Conversation

@shoumikhin

Copy link
Copy Markdown
Contributor

Supersedes #22204. Same change, plus the fix for a defect that made the new resource bundle empty for every real consumer.

What #22204 got wrong

The metallib resource list was gated on FileManager.default.fileExists with a path relative to the process working directory:

letmlxResourcesDir=".Package.swift/backend_mlx_resources"resources: mlxMetallibSlices.compactMap { slice inFileManager.default.fileExists(atPath:"\(mlxResourcesDir)/\(slice).metallib")?.copy("\(slice).metallib"):nil}

SwiftPM evaluates a dependency's manifest with the consumer's directory as the working directory, not the dependency's checkout. So every slice looked absent, compactMap returned [], and SwiftPM emitted no bundle at all. The build stayed green with no warning, and the MLX delegate would fault later at Metal device init inside a consumer's app with nothing pointing at the cause.

Measured from a real consumer package, with all three metallibs committed and present in the resolved checkout:

before: Build complete! bundles: 0 metallibs shipped: 0
after: Build complete! bundles: 1 metallibs shipped: 3
.build/out/Products/Debug/executorch_backend_mlx_resources.bundle

That bundle name is what backends/mlx/CMakeLists.txt compiles in as SWIFTPM_BUNDLE, so it is the name the delegate actually looks for.

Why it was easy to miss

Every in-place check passes. swift package dump-package run inside the package root reports all three resources, and --package-path chdirs to the package root too, so it is equally blind. Only resolving the package as a dependency exposes it. The release job's own guards check for unsubstituted checksums and for products without frameworks, neither of which can see this.

The fix

Declare the three slices unconditionally instead of probing for them. The release job commits all three files to this branch before publishing, so the probe was guarding against a state that does not occur on a published branch. import Foundation is no longer needed, and the base template did not have it.

What a genuinely missing slice does now, measured rather than assumed:

swift package resolve -> says nothing at all
swift build -> warning: 'dep': Invalid Resource 'mlx-ios.metallib': File not found
Build complete! (a warning, not an error)

So this makes the failure visible, not fatal. That is still a large improvement over the old form, which shipped zero of three metallibs with no diagnostic of any kind, but it is worth being precise: SwiftPM does not fail a build on a missing declared resource.

Landing

This half and the main-side half (#22203) both need to be live before the next Apple publish run. Order between them does not matter: each single-sided state trips one of the release job's two guards and exits 1 before git push -f, so nothing broken is published either way. Verified by replaying the substitution and both guards for all three states.

The published SwiftPM manifest is generated from this template, so the platform
floor a consumer sees is set here, not in the main branch's Package.swift.
Raising it to macOS 14 is what lets the MLX backend be offered through the
package. MLX requires a macOS 14 deployment target and fails to build below it,
so the package that ships MLX must declare at least that. iOS is already at 17,
which is above MLX's iOS floor, so only macOS moves.
This drops support for macOS 12 and 13 for the whole package, including consumers
that only use XNNPACK or Core ML. macOS 14 shipped in September 2023.
Test Plan:
Materialized the template the way the release job does, substituting the version
and checksum placeholders, and confirmed `swift package dump-package` parses it and
reports the macOS platform as 14.0.
Companion to the change that adds MLX to the Apple frameworks. The published
SwiftPM manifest is generated from this template, so the backend_mlx product and
its resource bundle have to be declared here too, or the released package would be
missing them.
Adds the backend_mlx product with its Metal, Foundation and QuartzCore framework
links, and a shared backend_mlx_resources target that carries the per-platform
Metal kernel libraries. The release job commits those libraries to this branch
beside the manifest. Each is included only when present so the manifest still
resolves before the release job has produced them.
The resource list was gated on FileManager.fileExists with a path relative to
the process working directory. SwiftPM evaluates a dependency's manifest with
the consumer's directory as the cwd, so every slice looked absent, the list came
out empty, and no bundle was produced at all. The build stayed green and the MLX
delegate faulted later at Metal device init, with nothing pointing at the cause.
Measured from a consumer package: 0 of 3 metallibs shipped before, 3 of 3 after,
in executorch_backend_mlx_resources.bundle, which is the name the delegate is
compiled to look for. An in-place build and swift package dump-package both
reported the resources correctly either way, which is why this was not caught.
The release job commits all three files before publishing, so declare them
unconditionally rather than probing for them. Foundation is no longer needed.
The comment said a missing slice is reported at resolve time. Measured: swift
package resolve emits nothing at all, and the "Invalid Resource ... File not
found" warning appears when a consumer builds. It is also only a warning, so the
build still succeeds; the change makes the failure visible, not fatal.
@pytorch-bot

pytorch-botBot commented Aug 28, 2026

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/22259

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 Unclassified Failure

As of commit 0e56f2b with merge base 1d467d0 (image):

UNCLASSIFIED FAILURE - DrCI could not classify the following job because the workflow did not run on the merge base. The failure may be pre-existing on trunk or introduced by this PR:

  • PR #22259 / Analyze (actions) (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
    CodeQL could not process any code written in GitHub Actions. For more information, review our troubleshooting guide at https://gh.io/troubleshooting-code-scanning/no-source-code-seen-during-build .

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-clameta-claBot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Aug 28, 2026
@shoumikhin

Copy link
Copy Markdown
ContributorAuthor

Superseded by a branch carrying two further review fixes: the comment no longer implies the main-side manifest is wrong for using FileManager.fileExists (it anchors with #filePath and is correct), and the single-use path local is inlined to match the sibling style. Reopening rather than leaving those unaddressed, since this repository blocks direct pushes to a branch once it has an open PR.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA SignedThis label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@shoumikhin