unified: Add Swift parser based on `swift-syntax` by tausbn · Pull Request #22195 · github/codeql

tausbn · 2026-07-15T11:32:13Z

Adds an alternative parser to tree-sitter-swift, using the Swift-native swift-syntax package instead.

The current implementation exposes this parser through a Rust wrapper, using the C FFI to communicate with Swift. The Rust component passes the source file contents to the Swift component, which in turn responds with a JSON string containing the AST.

Note that at present, the AST is quite verbose (though some effort has been made to make it less so). In particular, it contains detailed location information about every single token in the input, which is almost certainly overkill for our purposes.

This PR does not attempt to replace the existing yeast rules to be based on the swift-syntax AST. That is left as follow-up work.

Can be reviewed commit-by-commit.

Adds an initial prototype of an interface from Rust to Swift, which enables us to use the `swift-syntax` package for parsing. At present, the parsed AST is passed between Swift and Rust as a JSON string.

These were taking up roughly 20% of the JSON payload.

Adds a preliminary mapping that decodes the JSON into the yeast AST. The JSON AST itself is still very verbose, but it would be useful to see if it actually contains all of the things we want it to contain.

Introduces new yeast types for `Point`s and `Range`s (corresponding to the tree-sitter equivalents), since it seemed silly to have `swift-syntax-rs` pull in `tree-sitter` just to have those types available. This _does_ mean there is a slight overhead in the shared extractor when converting between these types, but I think this is negligible.

Also gathers these into a separate side-channel during the initial AST construction. That way, we don't encounter these as weird extra nodes while running yeast.

At this point it's only a proof-of-concept translation -- it translates `sourceFile` nodes, but everything else gets mapped to 'unsupported_node`.

Copilot

Pull request overview

Adds a Swift-native swift-syntax parser exposed through Rust and prepares unified extraction to consume its JSON AST.

Changes:

Adds Swift/Rust FFI parser and Bazel/Cargo builds.
Adds JSON-to-yeast AST adaptation and minimal Swift rules.
Extends yeast for externally constructed ASTs and owned locations.

Show a summary per file

File	Description
`MODULE.bazel`	Registers Swift dependencies and toolchains.
`Cargo.toml`	Adds the parser crate to the workspace.
`Cargo.lock`	Locks the new crate.
`shared/yeast-schema/src/schema.rs`	Supports merging schema names.
`shared/yeast-macros/src/parse.rs`	Uses yeast-owned ranges.
`shared/yeast/src/build.rs`	Migrates build contexts to owned ranges.
`shared/yeast/src/lib.rs`	Adds external AST construction/desugaring APIs.
`shared/yeast/src/range.rs`	Defines owned point and range types.
`shared/yeast/tests/test.rs`	Tests desugaring hand-built ASTs.
`shared/tree-sitter-extractor/src/extractor/mod.rs`	Converts yeast points for extraction.
`unified/extractor/src/languages/mod.rs`	Exposes the Swift JSON adapter.
`unified/extractor/src/languages/swift/adapter.rs`	Converts JSON into a yeast AST.
`unified/extractor/src/languages/swift/swift.rs`	Adds minimal swift-syntax rules.
`unified/extractor/tests/fixtures/let_x.swiftsyntax.json`	Provides parser output fixture.
`unified/extractor/tests/swift_syntax_pipeline.rs`	Tests the adapter/desugarer pipeline.
`unified/swift-syntax-rs/.gitignore`	Ignores generated outputs.
`unified/swift-syntax-rs/.swift-version`	Pins Swift 6.3.2.
`unified/swift-syntax-rs/BUILD.bazel`	Defines Bazel Swift/Rust targets.
`unified/swift-syntax-rs/Cargo.toml`	Defines the Rust crate and CLI.
`unified/swift-syntax-rs/README.md`	Documents building and usage.
`unified/swift-syntax-rs/build.rs`	Builds and links the Swift shim for Cargo.
`unified/swift-syntax-rs/src/lib.rs`	Implements safe Rust FFI bindings.
`unified/swift-syntax-rs/src/main.rs`	Adds the parser CLI.
`unified/swift-syntax-rs/swift/Package.resolved`	Locks swift-syntax.
`unified/swift-syntax-rs/swift/Package.swift`	Defines the Swift FFI package.
`unified/swift-syntax-rs/swift/Sources/SwiftSyntaxFFI/SwiftSyntaxFFI.swift`	Serializes SwiftSyntax ASTs as JSON.
`unified/swift-syntax-rs/xcode_transition.bzl`	Isolates macOS Xcode configuration.

Review details

Files reviewed: 26/27 changed files
Comments generated: 9
Review effort level: Medium

+ println!("cargo:rustc-link-search=native={}", build_dir.display());
+ println!("cargo:rustc-link-lib=dylib=SwiftSyntaxFFI");
+ println!("cargo:rustc-link-arg=-Wl,-rpath,{}", build_dir.display());


- BUILD.bazel: require x86_64 on the Linux branch of the compatibility select. The only registered standalone Linux Swift toolchain is x86_64-only, so without the CPU constraint Linux/aarch64 targets stayed "compatible" and then failed toolchain resolution instead of being skipped cleanly. - build.rs: emitting `rerun-if-changed` disables Cargo's default whole-package scan, so list every input to `swift build` — the package manifests, `Package.resolved`, `.swift-version`, and the whole Sources tree — so stale shim/toolchain output can't linger. (`.build/` is left unwatched, as it is this build's own output.) - src/lib.rs: a null result from the shim is not a parse failure — SwiftParser recovers from invalid syntax and always yields a tree — so it means the shim failed to serialize/allocate the JSON. Fix the error message and the variant doc accordingly. - README.md: `.swift-version` is not honored by the macOS Bazel build (which uses the host Xcode toolchain); document that it pins the Linux and local builds only. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The doc claimed the schema passed to `Ast::with_schema` must already have every node kind and field name registered, contradicting the registration methods just below and the swift-syntax adapter, which starts from `Schema::new()` and registers names on demand during construction. Document that up-front registration is optional. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

aCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

It seems that using swift_version_file has some issues when `codeql` is consumed as a dependency module. I'm hoping hardcoding the version instead will fix this, but ideally we should find a more robust solution.

Registers the `unified/swift-syntax-rs` workspace member (added in "unified: Add swift-syntax-rs") in the tree-sitter extractors crate universe. It has no external Rust dependencies, so its dependency entries are empty. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

jketema

I mostly skimmed the Rust changes.

For the Swift code I wonder whether we really need to write the complete serializer ourselves, or whether we could leverage things that are already there in swift-syntax. This is not blocking though.

Do we need someone with more Bazel knowledge to review those parts?

jketema · 2026-07-16T12:45:23Z

+If your `swift`/`swiftc` are not on `PATH`, point the build at them explicitly:
+
+```sh
+SWIFT=/path/to/swift SWIFTC=/path/to/swiftc cargo build
+```


Is it common to have something like this in rust builds that also compile code in other languages? Note that I'm not suggesting to change this.

I don't think we do it anywhere else, so it's a valid concern. Originally I had swift-syntax-rs much more tightly integrated with the extractor, in which case it made sense to build both at the same time. However, now that we have (for the time being) moved to a model where we call an external binary for the parsing, we could also decouple the builds themselves.

jketema · 2026-07-16T13:27:16Z

+//! swift-syntax JSON -> `swift_adapter::json_to_ast` -> yeast `Ast`
+//! -> `Desugarer::run_from_ast` (the real Swift translation rules) -> dump.
+//!
+//! This exercises the whole chain *without* the Swift toolchain: the JSON


This seems potentially fragile if swift-syntax changes?

Agreed. This was a "proof of concept" test that was useful at the time, but it is brittle. We can probably just get rid of it once the whole extraction is using swift-syntax-rs.

jketema

I think we should do the following (apologies for this being split in two comments).

Co-authored-by: Jeroen Ketema <93738568+jketema@users.noreply.github.com>

redsun82

Hey, thanks for this, seems like it was quite some work to get this working! I've looked at the bazel and build side of things, and didn't look at the actual compiled code at all. I don't have major comments on the overall direction, but I do have some comments I'd like answers to before we go ahead with this.

redsun82 · 2026-07-20T14:26:41Z

+ .current_dir(&swift_dir);
+ apply_bare_repository_workaround(&mut command);
+ let status = command.status().unwrap_or_else(|e| {
+ panic!(


could we make this degrade gracefully when there's no working swift toolchain, instead of hard-panicking? our actual builds go through bazel, and cargo day-to-day is mostly just fmt/check for us, but right now build.rs runs swift build unconditionally, so with no swift on PATH even those blow up. and since the crate is a workspace member now, a plain cargo check at the repo root fails for anyone who doesn't have swift installed, not just in here.
check/clippy don't link, so they don't actually need the swift lib to exist. so I think if swift build isn't runnable we could just emit a cargo:warning=... and skip the link directives rather than panic. that keeps the cargo fmt/check loop we rely on working swift-free, and only cargo build/cargo test would fail after that (at link, which is fair enough: those genuinely need swift, and we get them through bazel anyway).

(speaking from a Linux box perspective that doesn't have Swift preinstalled 😉)

I'll leave this one for @tausbn to fix.

I'll do this in the follow-up PR. It goes a long way towards totally decoupling the Swift build from the extractor build, so it'll need some extra attention at any rate.

redsun82

good to go for me from a build/bazel perspective. I'll leave you to decide whether #22195 (comment) should be done here or as a follow-up 🙂

tausbnand others added 8 commits July 10, 2026 11:35

unified: Add swift-syntax-rs
89e11e5
Adds an initial prototype of an interface from Rust to Swift, which enables us to use the `swift-syntax` package for parsing. At present, the parsed AST is passed between Swift and Rust as a JSON string.

unified: Don't emit empty trivia
87c8173
These were taking up roughly 20% of the JSON payload.

unified: Add mapping from swift-syntax JSON to yeast AST
446ff7c
Adds a preliminary mapping that decodes the JSON into the yeast AST. The JSON AST itself is still very verbose, but it would be useful to see if it actually contains all of the things we want it to contain.

unified: Emit locations for comments
ddab6cc
Also gathers these into a separate side-channel during the initial AST construction. That way, we don't encounter these as weird extra nodes while running yeast.

unified: Hook up swift-syntax AST
58ddeac
At this point it's only a proof-of-concept translation -- it translates `sourceFile` nodes, but everything else gets mapped to 'unsupported_node`.

Support building swift-syntax-rs on macOS
3e2daf2

Upgrade swift-syntax-rs to swift-syntax 603.0.2 / Swift 6.3.2
1577d82

github-actionsBot added the documentation label Jul 15, 2026

tausbn requested a review from Copilot July 15, 2026 11:32

Copilotstarted reviewing on behalf of tausbn July 15, 2026 11:33 View session

CopilotAI reviewed Jul 15, 2026
View reviewed changes

tausbnand others added 5 commits July 15, 2026 12:14

swift-syntax-rs: Fix bazel formatting
205c9a9
aCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

unified: Hardcode swift_version
7474a33
It seems that using swift_version_file has some issues when `codeql` is consumed as a dependency module. I'm hoping hardcoding the version instead will fix this, but ideally we should find a more robust solution.

tausbn force-pushed the tausbn/swift-syntax-rs branch from 4ba73bd to da1bbb7Compare July 15, 2026 13:15

tausbn marked this pull request as ready for review July 15, 2026 13:58

tausbn requested review from a team as code owners July 15, 2026 13:58

jketema previously approved these changes Jul 16, 2026
View reviewed changes

jketema reviewed Jul 17, 2026
View reviewed changes

Comment threadunified/swift-syntax-rs/BUILD.bazel Outdated
Comment threadunified/swift-syntax-rs/BUILD.bazel Outdated

unified: Clean up build
f384e29
Co-authored-by: Jeroen Ketema <93738568+jketema@users.noreply.github.com>

tausbn dismissed jketema’s stale review via f384e29July 20, 2026 11:10

redsun82 reviewed Jul 20, 2026
View reviewed changes

jketema added 3 commits July 21, 2026 10:49

Fix Bazel formatting
0a1139c

Simplify Xcode transition per review comments
bf57586

Only depend on the runtime libraries from the Swift toolchain
cf8eeb8

Use 22.04 Swift toolchain
38c9c84

redsun82 approved these changes Jul 22, 2026
View reviewed changes

tausbn merged commit a39eab3 into mainJul 23, 2026
147 of 150 checks passed

tausbn deleted the tausbn/swift-syntax-rs branch July 23, 2026 12:11

Uh oh!

Conversation

tausbn commented Jul 15, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

CopilotAI left a comment

Choose a reason for hiding this comment

Pull request overview

Review details

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jketema left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jketema left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

redsun82 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

redsun82 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

tausbn commented Jul 15, 2026•
edited
Loading