Skip to content

Optimize recursive sequence protocol rejection - #11695

Open
Bill Schnurr (bschnurr) wants to merge 7 commits into
microsoft:mainfrom
bschnurr:perf/recursive-sequence-protocol-fast-reject
Open

Optimize recursive sequence protocol rejection#11695
Bill Schnurr (bschnurr) wants to merge 7 commits into
microsoft:mainfrom
bschnurr:perf/recursive-sequence-protocol-fast-reject

Conversation

@bschnurr

@bschnurr Bill Schnurr (bschnurr) commented Sep 1, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Summary

Improve checking performance for lists of overloaded callables, including NumPy's np.array([np.sum, np.mean]) and pandas' test_series_apply.py, without adding NumPy-specific logic.

Recursive sequence rejection

  • Add a negative-only shortcut for built-in list assignments to a recognized recursive sequence protocol with one covariant type parameter. Tuples use ordinary protocol matching.
  • Require the specialized, non-overloaded __getitem__ return type to be exactly Leaf | Protocol[Leaf]. Similar member names alone are not sufficient.
  • Decline top-level source/destination element TypeVars, Any, and Unknown, and unsupported protocol shapes. This is not a recursive exclusion of all nested TypeVars or gradual types.
  • Clone caller constraints for the reduced element check. A successful reduced check does not establish protocol compatibility; ordinary structural matching still runs.

Constraint solving

  • Skip solveAndApplyConstraints when the target type requires no specialization and has no alias metadata, avoiding full constraint solves whose results would not transform the target.
  • Preserve the normal path for alias metadata because the transformer processes generic aliases before its ordinary no-op check.

Protocol cache provenance

  • Record whether a cached result came from fast rejection.
  • Allow a subsequent full check to replace the matching fast-rejection entry and exclude fast entries from the entry-existence check used when considering universal incompatibility.
  • This does not provide transactional cache isolation: cached negatives can still be reused before a diagnostic-bearing full check replaces them. Behavioral validation of inference and reuse remains an open gate below.

Problem and Correctness Argument

Why checking is slow

For np.array([np.sum, np.mean]), Pyright must determine which NumPy overload accepts a list of overloaded functions. Some candidates describe recursive sequences: each element can be a supported leaf value or another sequence. Rejecting an unsuitable candidate can trigger expensive recursive protocol matching, overloaded-function comparison, and repeated constraint solving. Profiling also identified substantial work solving constraints even when the particular type being transformed could not change.

The goal is to eliminate unnecessary work without rejecting valid candidates, accepting invalid ones, or losing inferred type precision. There are two independent changes: skip solving when substitution cannot transform the target (excluding alias metadata), and reject narrowly recognized recursive-sequence candidates using an element-compatibility condition. A successful element check never establishes full protocol compatibility; ordinary structural matching still runs.

Necessary-condition argument

Suppose the source is list[E] and protocol P[L] promises:

def __getitem__(self, index: int, /) -> L | P[L]: ...

For integer indexing, the built-in list signature returns E. Implementing the protocol therefore requires that this return type be assignable to the promised return type. Writing $\preceq$ for assignability:

$$ \mathrm{list}[E] \preceq P[L] \implies E \preceq L \cup P[L] $$

Taking the contrapositive gives the early-rejection rule:

$$ E \not\preceq L \cup P[L] \implies \mathrm{list}[E] \not\preceq P[L] $$

This justifies rejection, not acceptance: other protocol members can still be incompatible.

The integer-index premise is essential. A slice returns another list, not an element. A slice-only recursive protocol supplied a real counterexample to the return-type-only guard: the shortcut selected fallback str, whereas full matching selected Literal['slice']. A local follow-up adds a built-in-integer parameter guard and a precise slice regression. As of this description update (2026-09-04), that fix is in local commit ea13f4cc3, not the published PR head 34179d476. The published implementation therefore does not yet enforce this premise. The approximately 4.1-second post-fix pandas measurements and additional local tests must not be mistaken for validation of a published integer-index fix.

What an exception-free proof still requires

The implication above is not a complete proof of the implementation. The remaining obligations are:

  1. Eligibility establishes every premise. Binding, specialization, actual list signatures, parameter domains, and return types must have the meaning assumed by the argument.
  2. Reduced failure means genuine incompatibility. A false result must not merely mean that inference could not solve the relation in this context. Nested generics, conditional types, recursive assumptions, and incomplete inference matter; excluding only top-level Any, Unknown, and TypeVars does not prove this.
  3. Inference assumptions are appropriate. The reduced check must not reject a candidate that the full check could accept after solving relevant variables.
  4. The probe cannot corrupt later decisions. Cloning constraints protects the caller tracker, not every nested cache effect. Fast-rejection provenance prevents direct promotion to universal incompatibility, but is not a complete state-isolation proof.
  5. Skipping constraint solving is observationally harmless. Returning the same type is necessary; omitted solving must also have no required effects on subsequent inference. Alias metadata remains on the normal path because it can require transformation.

The proof target is that every early rejection is justified under the applicable typing rules and inference state, and subsequent decisions remain unaffected. Each obligation needs a code-backed argument or an explicit fallback when its premises cannot be established. Passing suites, adversarial cases, performance measurements, and no-shortcut comparisons provide evidence, not a universal guarantee. The discovered slice counterexample demonstrates the distinction. These obligations remain merge gates, not completed claims of correctness.

Performance

Latest local measurements used Node 26.5.0, Python 3.12.3, NumPy 2.4.6, and pandas revision 82a712a52a55e9ac99c59a818e36c4d400bc5dbe. The core was compiled before each CLI bundle. Base checker: c77393240247e017db6538d35876e723c518e06c; measured implementation revision: 9577c1663acf0751c858a8cd63d4b36c4145b911. The subsequent test-only commit 34179d47639970e3d8be13ca21949bde2af2f6a6 does not change production code; benchmarks have not been rerun for it.

Configuration Combined NumPy check time Combined NumPy wall time pandas file wall time
Base 79.75 s 80.69 s Historical: about 4 minutes
Solver guard alone, without sequence shortcut 24.68 s 25.66 s 78.53 s
Current integrated changes 0.27 s 1.22 s 4.18 s

The base singleton lists containing only np.sum or only np.mean checked in 0.32 and 0.34 seconds. The combined trigger revealed ndarray[tuple[Any, ...], dtype[Any]] with no errors or warnings across these measurements. The final pandas runs also reported no errors or warnings.

These are single-run local observations, not statistical benchmarks or Node 24 CI comparisons. The historical pandas base timing is not a fresh paired baseline. The independent solver guard provides a substantial partial improvement; it does not by itself retain the approximately four-second pandas result.

Tests and Local Validation

  • Existing sequence coverage includes positive nested sequences, incompatible callables, similar member names with different semantics, constrained generic results, tuple fallback, and subsequent valid list specializations after a rejection.
  • Direct helper tests cover scoped/unification TypeVars, Any/Unknown, concrete and generic-callable sources, and fast-cache replacement after a full diagnostic-bearing check.
  • Added SolveAndApplyConstraintsConcreteType: concrete types, Any, and Unknown avoid constraint-set traversal; bound TypeVars stay bound; free TypeVars specialize precisely through nested list[tuple[T, int]] and generic alias metadata.
  • Added eight test cases in 34179d476, plus expanded existing assertions:
    • Fresh evaluators exercise positive-first and negative-first protocol assignments, repeated rejection, diagnostic details, preserved caller constraints, and subsequent generic assignments inferring int with fresh constraint trackers.
    • A dependency-free overload sample asserts precise Literal['array'] selection for valid array-like sequences and str fallback for overloaded callables, scoped list[T], nested list[tuple[T, int]], ParamSpec callables, and TypeVarTuple tuples.
    • Uncertain destination elements (scoped/unification TypeVars, Any, Unknown) decline the helper without changing constraints.
    • Concrete unions, tuples, callables, None, and Never avoid solver traversal; alias assertions distinguish bound preservation from free-TypeVar substitution.
  • Full internal suite: 77 suites, 2,795 tests passed for the implementation preceding this test-only commit. The full suite has not been rerun after the new tests.
  • Latest pre-push targeted shard: 180 tests passed, with core TypeScript build, targeted ESLint/Prettier, and git diff --check also passing on 34179d476.
  • The earlier implementation validation also included a CLI development build. Jest emitted worker-exit/listener warnings; the latest targeted run emitted the force-exit notice.
  • No existing expected types or diagnostic expectations were weakened for performance.

Coverage Progress and Remaining Before Merge

  • Add focused protocol cache-reuse tests in both assignment orders, asserting results, caller constraints, diagnostic element incompatibility, and subsequent generic inference.
  • Validate broader inference-context and cross-candidate cache reuse beyond these focused scenarios; these tests do not establish general transactional cache isolation.
  • Add dependency-free recursive-protocol overload regressions for scoped list[T] and nested list[tuple[T, int]], callable fallback, and precise positive overload selection. These model the relevant matching behavior rather than importing NumPy.
  • Cover ParamSpec callables and TypeVarTuple tuple inputs in overload selection, and uncertain top-level destination elements in direct helper tests.
  • Add remaining nested Any/Unknown and incomplete-inference coverage at the shortcut boundary.
  • Repeat pandas measurements at least three times, targeting median wall time below five seconds in the pinned environment, and add stable small-trigger work/scaling coverage.
  • Validate the relevant hydra-zen, pandas-stubs, JAX, and frozen scikit-learn cases, then run the broader primer comparison with recorded revisions.
  • Resolve the reported SymPy dict[Dummy, Basic | Unknown] -> dict[Dummy, Basic] change with a reproducible, semantically justified expectation. Repeated unchanged-base full-project runs produced different diagnostics, so those broad diffs do not yet establish causality.

Correct typing behavior is the acceptance criterion, not exact parity with every old diagnostic. Any reproducible semantic change needs an explanation and focused coverage; passing local tests and having pushed the code do not close the remaining cache/inference gates.

@bschnurr

Bill Schnurr (bschnurr) commented Sep 1, 2026

Copy link
Copy Markdown
Member Author

/benchmark

@bschnurr
Bill Schnurr (bschnurr) force-pushed the perf/recursive-sequence-protocol-fast-reject branch from b214864 to a5239b7 Compare September 2, 2026 21:14
@github-actions

This comment has been minimized.

Sign in to view
@bschnurr
Bill Schnurr (bschnurr) force-pushed the perf/recursive-sequence-protocol-fast-reject branch from a5239b7 to 5118d44 Compare September 2, 2026 21:56
@github-actions

This comment has been minimized.

Sign in to view
@bschnurr
Bill Schnurr (bschnurr) force-pushed the perf/recursive-sequence-protocol-fast-reject branch from 5118d44 to b25be82 Compare September 2, 2026 22:42
@github-actions

This comment has been minimized.

Sign in to view
@bschnurr
Bill Schnurr (bschnurr) force-pushed the perf/recursive-sequence-protocol-fast-reject branch from b25be82 to 4f7b0ba Compare September 3, 2026 00:48
@github-actions

This comment has been minimized.

Sign in to view
@bschnurr
Bill Schnurr (bschnurr) force-pushed the perf/recursive-sequence-protocol-fast-reject branch from 4f7b0ba to e63dc1c Compare September 3, 2026 20:57
@github-actions

This comment has been minimized.

Sign in to view
@bschnurr
Defer recursive protocol rejection for unification TypeVars
06ed552 
Fall back to normal structural protocol matching when a list element is an unresolved unification TypeVar, and add regression coverage for the fast-path decision.
@github-actions

This comment has been minimized.

Sign in to view
@github-actions

This comment has been minimized.

Sign in to view
@github-actions

This comment has been minimized.

Sign in to view
root root
Optimize recursive protocol rejection inference
9577c16 
Skip constraint solving for types that cannot be specialized while preserving generic alias transformation. Track fast protocol-cache rejections separately so full checks can replace them and they cannot become universal incompatibility entries.\n\nFailure classification: B (Pyright performance limitation). The change preserves inferred type precision; focused tests cover concrete types, nested generics, aliases, and cache replacement.
@github-actions

This comment has been minimized.

Sign in to view
root root
Add regression coverage for protocol rejection and constraint solving
34179d4 
Cover cache reuse in both assignment orders, constraint preservation, generic overload fallback, concrete solver targets, and bound alias arguments.

Classification B: Pyright performance/inference coverage. No typeshed changes or weakened type expectations.
@github-actions

github-actions Bot commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Diff from mypy_primer, showing the effect of this PR on open source code:

sympy (https://github.com/sympy/sympy)
-   .../projects/sympy/sympy/solvers/bivariate.py:135:15 - error: Operator "-" not supported for "None" (reportOptionalOperand)
-   .../projects/sympy/sympy/solvers/bivariate.py:139:17 - error: Operator "-" not supported for type "Basic | Unknown" (reportOperatorIssue)
-   .../projects/sympy/sympy/solvers/bivariate.py:144:23 - error: Operator "-" not supported for "None" (reportOptionalOperand)
-   .../projects/sympy/sympy/solvers/deutils.py:234:14 - error: Operator "not in" not supported for types "str" and "Unknown | int"
-     Operator "not in" not supported for types "str" and "int" (reportOperatorIssue)
-   .../projects/sympy/sympy/solvers/diophantine/diophantine.py:423:38 - error: Argument of type "Unknown | Expr | Literal[0]" cannot be assigned to parameter "expr" of type "Expr" in function "make_args"
+   .../projects/sympy/sympy/solvers/diophantine/diophantine.py:423:38 - error: Argument of type "int | Expr" cannot be assigned to parameter "expr" of type "Expr" in function "make_args"
-     Type "Unknown | Expr | Literal[0]" is not assignable to type "Expr"
+     Type "int | Expr" is not assignable to type "Expr"
-       "Literal[0]" is not assignable to "Expr" (reportArgumentType)
+       "int" is not assignable to "Expr" (reportArgumentType)
+   .../projects/sympy/sympy/solvers/diophantine/diophantine.py:504:19 - error: Operator "**" not supported for types "Unknown | Basic" and "Literal[2]"
+     Operator "**" not supported for types "Basic" and "Literal[2]" (reportOperatorIssue)
+   .../projects/sympy/sympy/solvers/diophantine/diophantine.py:505:19 - error: Operator "*" not supported for types "Unknown | Basic" and "Unknown | Basic"
+     Operator "*" not supported for types "Basic" and "Basic" (reportOperatorIssue)
+   .../projects/sympy/sympy/solvers/diophantine/diophantine.py:506:19 - error: Operator "**" not supported for types "Unknown | Basic" and "Literal[2]"
+     Operator "**" not supported for types "Basic" and "Literal[2]" (reportOperatorIssue)
+   .../projects/sympy/sympy/solvers/diophantine/diophantine.py:569:42 - error: Operator "*" not supported for types "int" and "Unknown | Basic"
+     Operator "*" not supported for types "int" and "Basic" (reportOperatorIssue)
+   .../projects/sympy/sympy/solvers/diophantine/diophantine.py:569:50 - error: Operator "*" not supported for types "Expr" and "Unknown | Basic"
+     Operator "*" not supported for types "Expr" and "Basic" (reportOperatorIssue)
+   .../projects/sympy/sympy/solvers/diophantine/diophantine.py:725:42 - error: Operator "**" not supported for types "Unknown | Basic" and "Literal[2]"
+     Operator "**" not supported for types "Basic" and "Literal[2]" (reportOperatorIssue)
+   .../projects/sympy/sympy/solvers/diophantine/diophantine.py:735:19 - error: Operator "**" not supported for types "Unknown | Basic" and "Literal[2]"
+     Operator "**" not supported for types "Basic" and "Literal[2]" (reportOperatorIssue)
+   .../projects/sympy/sympy/solvers/diophantine/diophantine.py:736:19 - error: Operator "**" not supported for types "Unknown | Basic" and "Literal[2]"
+     Operator "**" not supported for types "Basic" and "Literal[2]" (reportOperatorIssue)
+   .../projects/sympy/sympy/solvers/diophantine/diophantine.py:737:19 - error: Operator "**" not supported for types "Unknown | Basic" and "Literal[2]"
+     Operator "**" not supported for types "Basic" and "Literal[2]" (reportOperatorIssue)
+   .../projects/sympy/sympy/solvers/diophantine/diophantine.py:814:47 - error: Operator "**" not supported for types "Unknown | Basic" and "Literal[2]"
+     Operator "**" not supported for types "Basic" and "Literal[2]" (reportOperatorIssue)
+   .../projects/sympy/sympy/solvers/diophantine/diophantine.py:841:26 - error: Operator "**" not supported for types "Unknown | Basic" and "Literal[2]"
+     Operator "**" not supported for types "Basic" and "Literal[2]" (reportOperatorIssue)
+   .../projects/sympy/sympy/solvers/diophantine/diophantine.py:842:22 - error: Operator "*" not supported for types "Unknown | Basic" and "Unknown | Basic"
+     Operator "*" not supported for types "Basic" and "Basic" (reportOperatorIssue)
+   .../projects/sympy/sympy/solvers/diophantine/diophantine.py:843:36 - error: Operator "*" not supported for types "int" and "Unknown | Basic"
+     Operator "*" not supported for types "int" and "Basic" (reportOperatorIssue)
+   .../projects/sympy/sympy/solvers/diophantine/diophantine.py:843:42 - error: Operator "*" not supported for types "Unknown | Basic" and "Unknown | Basic"
+     Operator "*" not supported for types "Basic" and "Basic" (reportOperatorIssue)
+   .../projects/sympy/sympy/solvers/diophantine/diophantine.py:843:51 - error: Operator "*" not supported for types "int" and "Unknown | Basic"
+     Operator "*" not supported for types "int" and "Basic" (reportOperatorIssue)
+   .../projects/sympy/sympy/solvers/diophantine/diophantine.py:843:57 - error: Operator "*" not supported for types "Unknown | Basic" and "Unknown | Basic"
+     Operator "*" not supported for types "Basic" and "Basic" (reportOperatorIssue)
+   .../projects/sympy/sympy/solvers/diophantine/diophantine.py:843:66 - error: Operator "*" not supported for types "Unknown | Basic" and "Unknown | Basic"
+     Operator "*" not supported for types "Basic" and "Basic" (reportOperatorIssue)
+   .../projects/sympy/sympy/solvers/diophantine/diophantine.py:845:53 - error: Operator "*" not supported for types "Unknown | Basic" and "Unknown | Basic"
+     Operator "*" not supported for types "Basic" and "Basic" (reportOperatorIssue)
+   .../projects/sympy/sympy/solvers/diophantine/diophantine.py:854:18 - error: Operator "**" not supported for types "Unknown | Basic" and "Literal[2]"
+     Operator "**" not supported for types "Basic" and "Literal[2]" (reportOperatorIssue)
+   .../projects/sympy/sympy/solvers/diophantine/diophantine.py:856:22 - error: Operator "**" not supported for types "Unknown | Basic" and "Literal[2]"
+     Operator "**" not supported for types "Basic" and "Literal[2]" (reportOperatorIssue)
+   .../projects/sympy/sympy/solvers/diophantine/diophantine.py:865:22 - error: Operator "*" not supported for types "Unknown | Basic" and "Unknown | Basic"
+     Operator "*" not supported for types "Basic" and "Basic" (reportOperatorIssue)
+   .../projects/sympy/sympy/solvers/diophantine/diophantine.py:865:36 - error: Operator "*" not supported for types "Unknown | Basic" and "Unknown | Basic"
+     Operator "*" not supported for types "Basic" and "Basic" (reportOperatorIssue)
+   .../projects/sympy/sympy/solvers/diophantine/diophantine.py:867:27 - error: Operator "**" not supported for types "Unknown | Basic" and "Literal[2]"
+     Operator "**" not supported for types "Basic" and "Literal[2]" (reportOperatorIssue)
+   .../projects/sympy/sympy/solvers/diophantine/diophantine.py:868:27 - error: Operator "*" not supported for types "Unknown | Basic" and "Unknown | Basic"
+     Operator "*" not supported for types "Basic" and "Basic" (reportOperatorIssue)
+   .../projects/sympy/sympy/solvers/diophantine/diophantine.py:869:27 - error: Operator "*" not supported for types "Unknown | Basic" and "Unknown | Basic"
+     Operator "*" not supported for types "Basic" and "Basic" (reportOperatorIssue)
+   .../projects/sympy/sympy/solvers/diophantine/diophantine.py:870:27 - error: Operator "**" not supported for types "Unknown | Basic" and "Literal[2]"
+     Operator "**" not supported for types "Basic" and "Literal[2]" (reportOperatorIssue)
+   .../projects/sympy/sympy/solvers/diophantine/diophantine.py:871:27 - error: Operator "*" not supported for types "Unknown | Basic" and "Unknown | Basic"
+     Operator "*" not supported for types "Basic" and "Basic" (reportOperatorIssue)
+   .../projects/sympy/sympy/solvers/diophantine/diophantine.py:872:27 - error: Operator "**" not supported for types "Unknown | Basic" and "Literal[2]"
+     Operator "**" not supported for types "Basic" and "Literal[2]" (reportOperatorIssue)
+   .../projects/sympy/sympy/solvers/diophantine/diophantine.py:876:24 - error: Operator "**" not supported for types "Unknown | Basic" and "Literal[2]"
+     Operator "**" not supported for types "Basic" and "Literal[2]" (reportOperatorIssue)
+   .../projects/sympy/sympy/solvers/diophantine/diophantine.py:877:24 - error: Operator "**" not supported for types "Unknown | Basic" and "Literal[2]"
+     Operator "**" not supported for types "Basic" and "Literal[2]" (reportOperatorIssue)
+   .../projects/sympy/sympy/solvers/diophantine/diophantine.py:878:24 - error: Operator "**" not supported for types "Unknown | Basic" and "Literal[2]"
+     Operator "**" not supported for types "Basic" and "Literal[2]" (reportOperatorIssue)
+   .../projects/sympy/sympy/solvers/diophantine/diophantine.py:879:24 - error: Operator "*" not supported for types "Unknown | Basic" and "Unknown | Basic"
+     Operator "*" not supported for types "Basic" and "Basic" (reportOperatorIssue)
+   .../projects/sympy/sympy/solvers/diophantine/diophantine.py:880:24 - error: Operator "*" not supported for types "Unknown | Basic" and "Unknown | Basic"
+     Operator "*" not supported for types "Basic" and "Basic" (reportOperatorIssue)
+   .../projects/sympy/sympy/solvers/diophantine/diophantine.py:881:24 - error: Operator "*" not supported for types "Unknown | Basic" and "Unknown | Basic"
+     Operator "*" not supported for types "Basic" and "Basic" (reportOperatorIssue)
+   .../projects/sympy/sympy/solvers/diophantine/diophantine.py:888:37 - error: Operator "*" not supported for types "int" and "Unknown | None"
+     Operator "*" not supported for types "int" and "None" (reportOperatorIssue)
+   .../projects/sympy/sympy/solvers/diophantine/diophantine.py:888:45 - error: Operator "*" not supported for types "int" and "Unknown | None"
+     Operator "*" not supported for types "int" and "None" (reportOperatorIssue)

... (truncated 1745 lines) ...

@bschnurr Bill Schnurr (bschnurr) mentioned this pull request Sep 8, 2026
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@bschnurr