Uh oh!
There was an error while loading. Please reload this page.
gh-153044: Auto-possessify greedy repeats with a disjoint continuation - #153048
Open
serhiy-storchaka wants to merge 5 commits into
Open
gh-153044: Auto-possessify greedy repeats with a disjoint continuation#153048serhiy-storchaka wants to merge 5 commits into
serhiy-storchaka wants to merge 5 commits into
Conversation
…nuation
Add an optimizer pass that turns a greedy repeat into a possessive one
when this cannot change what the pattern matches: if the repeated atom
and every character that could follow the repeat are disjoint,
backtracking into the repeat is always futile. For example "a+b" is
compiled as if it were "a++b". PCRE2 calls this "auto-possessification".
The analysis only proves disjointness, so any imprecision can merely
cost an optimization, never change a match result. It speeds up
backtracking-heavy failures whose tail is a character set or category
(not covered by the REPEAT_ONE fast path) severalfold and defuses some
catastrophic backtracking.
Category atoms are decided by the engine itself through a new helper,
_sre.category_matches(), so every engine category is decided exactly:
\p{Lu}+x possessifies, and a category and its complement are always
disjoint, so \p{X}+\P{X} possessifies for every X. Under IGNORECASE
only the fold-invariant categories keep their meaning, and the analysis
evaluates a fused set-operation charset exactly, so [\w--\d]+\d and
[a-z--b]+b possessify too.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>Documentation build overview
|
Unicode \w and ascii \W overlap, so a category and its complement are only disjoint within one flag context; this holds because the walk never compares atoms across a flag-scoping boundary. Add tests pinning that (?a:\w+)\W and \w+(?a:\W) are not possessified, while \w+\W and (?a:\w+\W) are. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Each empty branch alternative appends the continuation to the follower set again, so patterns like (|)(|)(|)... made the analysis exponential in time and memory (found by OSS-Fuzz). Give up when the set exceeds a small limit; a repeat with dozens of distinct followers is not going to be proven disjoint anyway. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The follower scan recurses once per following group, so thousands of sequential groups overflowed the recursion limit (the parser handles them iteratively). Add an explicit depth cap -- relying on RecursionError is fragile near the limit -- and keep catching it as a backstop: every rewrite already applied is sound on its own, so the rest of the pattern just stays unoptimized. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
| return av[0][1] | ||
| return None | ||
| def _disjoint(atom, other, flags): |
Contributor
There was a problem hiding this comment.
The PR increases the compilation time for regexes, about 20% on the ones I tested. In particular I have tested the _RATIONAL_FORMAT from fractions.py
Line 56 in 9cbb9e5
to see whether this PR works as an alternative to #149024.
According to Claude _disjoint is one of the hotspots and can be improved with a lru cache (other improvements might be possible as well)
| data[:] = out | ||
| def _walk(seq): | ||
| def _walk(seq, flags, cont): |
Contributor
There was a problem hiding this comment.
Even tough it is no public interface, can we document the new cont argument?
| if flags & SRE_FLAG_LOCALE or flags & SRE_FLAG_UNICODE: | ||
| return None # Unicode/locale categories are not small | ||
| if iav is CATEGORY_DIGIT: | ||
| out.update(range(0x30, 0x3a)) |
Contributor
There was a problem hiding this comment.
Can we make this a constant _ASCII_DIGIT just like _ASCII_WORD?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add an optimizer pass that turns a greedy repeat into a possessive one when this cannot change what the pattern matches: if the repeated atom and every character that could possibly follow the repeat are provably disjoint, backtracking into the repeat is always futile, so
a+bis compiled as if it werea++b. PCRE2 performs the same optimization under the name "auto-possessification".The analysis only ever fires when it can prove disjointness, so any imprecision can merely cost an optimization, never change a match result. The repeated body may be a single one-character atom or a rigid concatenation (only its leading atom matters); the follower analysis walks through group boundaries, alternations, nullable repeats, atomic groups and the
\z/$anchors. Disjointness is proved by an atom model for\d/\w/\s, by finite witness sets (case-folded under IGNORECASE), or by probing the engine's own category predicate through a new helper_sre.category_matches(), so every engine category is decided exactly:\p{Lu}+xpossessifies, and a category and its complement are always disjoint, so\p{X}+\P{X}possessifies for every X. A fused set-operation charset is evaluated exactly, so[\w--\d]+\dand[a-z--b]+bpossessify too.The main benefit is failing or backtracking-heavy matches whose repeat is followed by a character set or category, which the existing REPEAT_ONE literal fast path does not cover: measured 2.2–4.5× on such failing matches,
and some catastrophic-backtracking patterns are defused as a side effect. Successful matches are unaffected (~1×). Correction: where disjointness is provable, every give-back already fails at the first following atom, so this removes only a constant factor; the catastrophic nested-repeat patterns like(a+)+bare among the possible extensions.Validated by differential fuzzing against the unoptimized compilation (millions of checks accumulated across pattern families: random ASCII, IGNORECASE with non-ASCII fold groups, scoped flags, anchors, set operations,
\p{...}categories, exhaustive small-alphabet brute force), plus full-range exactness checks of the category probes against the engine.This is the first step described in the issue; alternation bodies with disjoint first characters, word-boundary and lookahead followers, and category-vs-category reasoning are possible extensions.