Skip to content

Add tree-sitter-swift extractor scaffolding and YEAST desugaring - #21812

Merged
asgerf merged 7 commits into
github:mainfrom
asgerf:asgerf/swift-yeast-1
May 8, 2026
Merged

Add tree-sitter-swift extractor scaffolding and YEAST desugaring#21812
asgerf merged 7 commits into
github:mainfrom
asgerf:asgerf/swift-yeast-1

Conversation

@asgerf

@asgerfasgerf commented May 7, 2026

Copy link
Copy Markdown
Contributor
  • Adds boilerplate for a tree-sitter based extractor using tree-sitter-swift.
  • Uses YEAST to perform a dummy desugaring.
  • A CodeQL test verifies that the desugaring is happening.
  • Adds AGENTS.md to help with worktrees

The extractor is placed in a folder called unified because the intent is to have more languages in there eventually.

Some follow-up work I have not included in the PR but working on:

  • Corpus-style extractor tests with schema validation
  • Changing the output schema

Comment threadunified/ql/lib/codeql/Locations.qll Dismissed
@asgerf

asgerf commented May 7, 2026

Copy link
Copy Markdown
ContributorAuthor
Rerun has been triggered: 1 restarted 🚀

@asgerfasgerf added the no-change-note-required This PR does not need a change note label May 7, 2026
@asgerf
asgerfforce-pushed the asgerf/swift-yeast-1 branch from 5011bfc to 2802819CompareMay 7, 2026 19:38
@asgerf

asgerf commented May 7, 2026

Copy link
Copy Markdown
ContributorAuthor
Rerun has been triggered, but no failed runs found ⚠️

@asgerf
asgerf marked this pull request as ready for review May 7, 2026 20:18
CopilotAI review requested due to automatic review settings May 7, 2026 20:18
@asgerf
asgerf requested review from a team as code ownersMay 7, 2026 20:18
@asgerf
asgerf requested a review from tausbnMay 7, 2026 20:19

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces an initial “unified” tree-sitter-based extractor scaffold (currently targeting Swift via tree-sitter-swift), wires in a simple YEAST desugaring pass, and adds a small CodeQL library-test to confirm the desugaring is applied during extraction.

Changes:

  • Add a new Rust-based unified extractor binary with extract, generate, and autobuild subcommands.
  • Add packaging/build integration (Bazel + Cargo workspace) and helper scripts for local pack generation and qltest indexing.
  • Add a basic unified QL pack/test pack and a Swift library-test verifying the desugaring effect.
Show a summary per file
FileDescription
unified/tools/qltest.shAdds qltest indexing for Swift-related extensions in unified tests.
unified/tools/qltest.cmdWindows equivalent qltest indexing script for unified.
unified/tools/index-files.shHooks CodeQL “index-files” into the unified extractor’s extract subcommand.
unified/tools/index-files.cmdWindows equivalent for unified index-files.
unified/tools/BUILD.bazelPackages unified tool scripts into the extractor pack.
unified/tools/autobuild.shRuns unified extractor autobuild.
unified/tools/autobuild.cmdWindows equivalent for unified autobuild.
unified/scripts/create-extractor-pack.shLocal script to build and assemble an extractor pack for testing.
unified/ql/test/qlpack.ymlDefines the unified test pack and its workspace dependencies.
unified/ql/test/qlpack.lock.ymlLockfile for unified test pack.
unified/ql/test/library-tests/BasicTest/test.swiftSwift input used by the library-test.
unified/ql/test/library-tests/BasicTest/test.qlLibrary-test query predicates validating AST/desugaring behavior.
unified/ql/test/library-tests/BasicTest/test.expectedExpected results validating the desugaring effect.
unified/ql/src/qlpack.ymlDefines the unified queries pack.
unified/ql/src/qlpack.lock.ymlLockfile for unified queries pack.
unified/ql/lib/unified.dbscheme.statsInitial dbscheme stats file for unified.
unified/ql/lib/qlpack.ymlDefines the unified library pack and dbscheme wiring.
unified/ql/lib/qlpack.lock.ymlLockfile for unified library pack.
unified/ql/lib/codeql/unified/Ast.qllGenerated Swift AST library produced from tree-sitter node types.
unified/ql/lib/codeql/Locations.qllAdds a minimal locations helper library for unified QL.
unified/ql/lib/codeql/files/FileSystem.qllAdds minimal file system abstractions for unified QL.
unified/ql/lib/BUILD.bazelBazel packaging targets for unified dbscheme artifacts.
unified/extractor/src/main.rsCLI entrypoint wiring extract/generate/autobuild.
unified/extractor/src/languages/swift/swift.rsSwift language spec, including a YEAST desugaring phase.
unified/extractor/src/generator.rsGenerates dbscheme/QL library for the Swift grammar.
unified/extractor/src/extractor.rsUnified extractor implementation using the shared tree-sitter extractor framework.
unified/extractor/src/autobuilder.rsUnified autobuilder wrapper around shared autobuilder support.
unified/extractor/Cargo.tomlNew crate definition and dependencies for unified extractor.
unified/extractor/BUILD.bazelBazel build rule for the unified Rust extractor binary.
unified/codeql-extractor.ymlDeclares the new unified extractor metadata and file types.
unified/BUILD.bazelBazel packaging rule to assemble the unified extractor pack.
unified/AGENTS.mdContributor instructions for building/testing the unified extractor.
unified/.gitignoreIgnores local build/test outputs for unified.
MODULE.bazelAdds tree-sitter-swift vendored repo to Bazel module deps.
misc/bazel/3rdparty/tree_sitter_extractors_deps/defs.bzlAdds unified extractor crate dependency mapping and swift crate repo.
misc/bazel/3rdparty/tree_sitter_extractors_deps/BUILD.tree-sitter-swift-0.7.2.bazelGenerated Bazel BUILD for tree-sitter-swift crate.
misc/bazel/3rdparty/tree_sitter_extractors_deps/BUILD.bazelAdds aliases for the tree-sitter-swift crate.
Cargo.tomlAdds unified/extractor to the workspace members.
Cargo.lockAdds lock entries for codeql-extractor-unified and tree-sitter-swift.

Copilot's findings

  • Files reviewed: 37/40 changed files
  • Comments generated: 6

Comment threadunified/extractor/BUILD.bazel Outdated
Comment threadunified/extractor/src/extractor.rs Outdated
Comment threadunified/extractor/src/extractor.rs Outdated
Comment threadunified/extractor/src/extractor.rs Outdated
Comment threadunified/extractor/src/generator.rs Outdated
Comment threadunified/extractor/src/generator.rs Outdated
asgerfand others added 2 commits May 8, 2026 08:56
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
@asgerf

asgerf commented May 8, 2026

Copy link
Copy Markdown
ContributorAuthor
Rerun has been triggered: 1 restarted 🚀

@hvitvedhvitved left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks plausible to me.

@tausbntausbn left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks reasonable to me. Let's get it in.

Comment threadunified/AGENTS.md
@asgerf
asgerf merged commit fdef477 into github:mainMay 8, 2026
84 checks passed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentationno-change-note-requiredThis PR does not need a change note

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants

@asgerf@tausbn@hvitved@github-advanced-security