diffract

An OCaml library and CLI tool for parsing source files using tree-sitter and pattern matching with concrete syntax.

Features

• Parse source using tree-sitter grammars
• Pattern matching with concrete syntax and metavariables
• Generic patch transforms in the style of Coccinelle (not-really semantic patches)
• Expansion transforms: join or restructure each element of a matched sequence
• Change summaries (summarize): infer the spatch rules behind a changeset — cluster the systematic edits in a before/after directory pair into rules, with per-file residuals for everything the rules don't explain
• Support for TypeScript, TSX, Kotlin, PHP, Scala (extensible)

Building

Prerequisites

• OCaml 5.4+
• opam
• tree-sitter library (libtree-sitter) - see below
• npm (to fetch grammar sources)

Installing tree-sitter

macOS:

brew install tree-sitter

Ubuntu/Debian:

sudo apt install libtree-sitter-dev

Arch Linux:

sudo pacman -S tree-sitter

Build Steps

# Install OCaml dependencies (add --with-test to include test deps)
opam install . --deps-only --with-test
# Build grammar libraries (TypeScript, Kotlin)cd grammars && ./build-grammars.sh &&cd ..
# Build the project
dune build
# Run tests
dune test# Format code (requires ocamlformat: opam install ocamlformat)
dune fmt
# Enable the pre-push hook (builds & tests the pushed commit in a clean# worktree, so untracked files can't mask a broken commit)
git config core.hooksPath .githooks

Formatting is managed via ocamlformat. Run dune fmt to reformat all OCaml and dune files before committing.

Because of caching, rerunning dune fmt might not produce warnings for things like stray @ in doc comments. One can just run ocamlformat manually:

ocamlformat --check $(find lib -name '*.ml' -o -name '*.mli')

Usage

# Parse and print the syntax tree
diffract parse example.ts
diffract parse --language kotlin example.kt
# Search for a pattern in a single file
diffract search pattern.txt source.ts
# Scan a directory for pattern matches
diffract search --include '*.ts' pattern.txt src/
# Scan with custom directory exclusions
diffract search --include '*.ts' -e vendor -e dist pattern.txt src/
# Show how a pattern tokenizes (diagnose a pattern that matches nothing)
diffract search --debug-tokens pattern.txt source.ts
# Apply a semantic patch (preview diff)
diffract apply patch.txt source.ts
# Apply a semantic patch in place
diffract apply --in-place patch.txt source.ts
# Apply across a directory
diffract apply --include '*.ts' patch.txt src/
# Show AST-level changes between two file versions
diffract diff before.ts after.ts
# Summarize a changeset: infer the rules behind a before/after directory pair
diffract summarize -l typescript -i '*.ts' before/ after/
# List available languages
diffract languages

Transforms (Semantic Patches)

Patterns can include -/+ prefixed lines to describe code transformations. For example, to rename console.log to logger.info:

patch.txt:

@@
match: strict
metavar $MSG: single
@@
- console.log($MSG)
+ logger.info($MSG)

$ diffract apply patch.txt source.ts
--- a/source.ts
+++ b/source.ts
@@ -1,3 +1,3 @@
functiongreet(name: string) {
- console.log(name);
+ logger.info(name);
}

Lines prefixed with - are matched and removed; lines with + are inserted. Unprefixed (or space-prefixed) lines are context that appears in both match and replace. Metavariables carry values from the match side to the replace side.

Sequence rendering: join and foreach

A sequence metavar referenced in a + template is rendered and substituted in place. Two knobs: a join $VAR by "<sep>" preamble directive sets the separator between elements (default: empty), and a following foreach $VAR section applies a per-element transform.

For per-element transforms (e.g. converting a match expression to a method chain), use a two-section pattern with foreach:

@@
match: strict
metavar $TAG: single
metavar $CASES: sequence
@@
- matchExhaustive($TAG, { $CASES });
+ match($TAG)$CASES.exhaustive();
@@
match: strict
foreach $CASES
metavar $KEY: single
metavar $VAL: single
@@
- $KEY: $VAL
+ .with("$KEY", $VAL)

Applied to:

matchExhaustive(tag,{A: ()=>1,B: ()=>2});

Produces:

match(tag).with("A",()=>1).with("B",()=>2).exhaustive();

See Transform documentation for partial-mode, field-mode, and sequence transforms.

Change Summaries (summarize)

summarize runs the transform machinery in reverse: given a before/after pair of directory trees, it infers the semantic-patch rules behind the changeset. Instead of reading the same edit repeated across a large diff, a reviewer reads each rule once and then inspects only the residuals — per-file diffs of whatever no rule explains. For example, a changeset that converts Kotlin block bodies to expression bodies:

// before/a.kt // after/a.ktfunlabel(): String { funlabel(): String="repo"return"repo"
}
overridefuncount(): Int { overridefuncount(): Int= items.size
return items.size
}

$ diffract summarize -l kotlin -i '*.kt' before/ after/
# rule R1 support=6 language=kotlin
@@
match: field
metavar _H0: single
metavar _H1: single
@@
fun _H1(...)
- {
- return _H0
- }
+ = _H0
# sites R1
a.kt
b.kt

One match: field rule covers every site even though the signatures differ — visibility modifiers, override, return types, parameter lists — because field mode aligns only the parts the pattern addresses. And the rules are safe: a rule claims a file only when applying it there is verified to reproduce part of that file's actual change, so look-alike sites that did something subtly different are reported as residuals instead of being claimed wrongly.

On a real-world changeset this example is distilled from -- a mechanical refactor touching ~100 Kotlin files, ~2,700 changed diff lines -— summarize reduces the whole diff to two rules plus two small residuals.

See Change summaries for the output format, more worked examples, and --ignore-formatting.

Directory Scanning

When the target is a directory, use --include to specify which files to scan:

Option	Description
--include GLOB / -i	Glob pattern for files (e.g., *.ts, *.py). Required for directories.
--exclude DIR / -e	Directory names to skip (repeatable). Defaults: node_modules, .git, _build, target, __pycache__, .hg, .svn

Supported glob patterns:

• *.ts - files ending with .ts
• prefix* - files starting with prefix
• *suffix - files ending with suffix

Example output:

src/api/auth.ts:15: console.log("login")
$msg = "login"
src/utils/logger.ts:8: console.log("initialized")
$msg = "initialized"
Found 2 match(es) in 2 file(s) (scanned 47 files)

Documentation

User guides:

• Pattern format and library API
• Transforms (semantic patches)
• Partial mode, field mode, surgical transforms
• Change summaries (summarize)

Architecture and design:

• Architecture and internals
• The universal tokenizer matcher (design note)
• Change-summary design (safety property, propose/evaluate/select, the geodesic gate, tiers)
• Grammar metadata
• References / existing work — the papers and ideas the implementation draws on, and where each is used
• Testing, migration coverage

Matcher Architecture (Quick Overview)

The matching pipeline is split into focused modules:

• tokenize parses a pattern body with tree-sitter as a lexer, keeping leaves as a (text, node_type) token stream (sigil-free metavars, ellipsis, fragments).
• cursor / tree_sitter_cursor define and implement the tree-cursor interface the engine runs over.
• stmatch is the matching engine: leaf-level strict / partial / field matching with backtracking, over any Cursor.S.
• matcher ties it together — preamble parse → tokenize → match → transform — and exposes the public find / transform / debug_tokens / pattern_warnings API.

License

GPL-3.0-or-later