Python: switch dataflow library to new (shared) CFG + SSA by yoff · Pull Request #21925 · github/codeql

yoff · 2026-06-01T12:29:13Z

Summary

The trunk-flip equivalent of the original draft PR #21894 (kept around as documentation of the end state), rebased on top of the four preparatory PRs in this stack:

P1: #21919 — Remove AstNode.getAFlowNode() and rewrite callers.
P2: #21920 — Qualify Flow.qll's AST references with Py:: prefix.
P3: #21921 — Add new shared-CFG-backed control flow graph (additive).
P4: #21923 — Add new shared-SSA-backed SSA adapter (additive).

Based on #21923 — merge P1–P4 first.

Flips the Python dataflow trunk from the legacy CFG (semmle/python/Flow.qll) and legacy ESSA SSA (semmle/python/essa/*) to the new shared CFG facade (semmle.python.controlflow.internal.Cfg) and the new SSA adapter (semmle.python.dataflow.new.internal.SsaImpl).

What changes

Dataflow library

The Python dataflow library (semmle/python/dataflow/new/) now imports the new CFG facade and SSA adapter. All CFG-typed predicates (ControlFlowNode, CallNode, BasicBlock, NameNode, AttrNode, ...) are qualified with the Cfg:: prefix; SSA references switch from EssaVariable/EssaDefinition to SsaImpl::Definition/SourceVariable.

GuardNode redesign

GuardNode is redesigned to use the new CFG's outcome-node model (isAfterTrue/isAfterFalse) instead of the legacy ConditionBlock + flipped indirection. Only BarrierGuard<...> is preserved as public API — the rest of the legacy GuardNode surface (isSafeCheck, flipped) was Python-specific and had no callers outside this library.

Framework updates

Framework files (Bottle, FastApi, Django, Tornado, Pyramid, Stdlib, MarkupSafe, Pycurl, Pydantic, Gradio, ...) are updated to take CFG nodes from the new facade.

Dataflow consistency tweaks for the new CFG

Augmented-assignment targets are treated as both load and store.
from X import * produces uncertain SSA writes for unknown names.
CFG nodes are canonicalised so dataflow does not see equivalent pre/post-order pairs as distinct nodes.

Two AST tweaks for the new CFG

AstNodeImpl: omit PEP 695 type-parameter names from FunctionDefExpr/ClassDefExpr children (they belong to the type-params block, not the function/class header).
ImportResolution: drop the legacy essa import.

Test churn (~135 reblessed `.expected`)

The reblessed .expected files fall into two buckets:

Cosmetic node toString relabel (the dominant change). The new CFG node prints as X (or After X) where the legacy CFG printed ControlFlowNode for X. 118 of the 135 reblessed .expected are identical to the legacy output once this relabel is normalised away — i.e. a pure, alert-preserving rename.
Slightly different CFG granularity. Flow-node-counting library tests (def-use-flow, use-use-flow, coverage, typetracking) see different counts because the new CFG has separate pre/post nodes per expression. These are node-count deltas, not alert deltas.

Alert preservation — investigation

To confirm the flip preserves results (not just node labels), the 135 reblessed .expected were audited against the legacy (pre-flip) output, normalising the node-toString relabel:

118 / 135 are byte-identical to legacy modulo the relabel — no result change at all.
17 / 135 differ in content. Each was inspected individually; all are either node-granularity count tests (above) or genuine precision improvements, namely:
- sqlalchemy / stdlib concept tests: more Concepts resolved (+20, +4).
- CallGraph type-tracking: calls that legacy could only resolve via points-to are now also resolved by type-tracking (MISSING: tt=… annotations removed).
- RequestWithoutValidation: the same alerts, with a richer message tracing verify=False back to its origin (… because $@ by $@).
- UnsafeUnpacking: identical #select; only an extra provenance hop in the path graph.
No security alert is lost. Every #select/problem row present in the legacy output is present after the flip.

Independent corroboration: 86 of the reblessed tests carry inline // $/# $ expectation annotations (via postprocess: utils/test/InlineExpectationsTestQuery.ql). Those annotations still match, witnessing preservation directly at the alert level rather than via the snapshot.

Snapshot-freshness fix

While auditing, a subset of the flip's .expected were found to be stale — they had been blessed at an earlier point in the stack, before the canonical (injects-only) ControlFlowNode representative and later dataflow refinements landed in the preparatory PRs, and had not been refreshed. The most visible symptom was CWE-022-PathInjection, whose stale snapshot recorded a Missing result: Source for a FastAPI Depends() parameter that the current code in fact reports correctly. All affected dirs were re-blessed; 51.expected were refreshed (the rest already matched), and the audit above was run on the refreshed result.

Verification

All 367 lib/ + src/ + consistency-queries/ queries compile clean against the new trunk.
All ControlFlow + PointsTo + dataflow + dataflow-new-ssa + essa + consistency library-tests pass, and the reblessed query-tests pass without --learn.
CFG consistency verified at scale: consistency-queries/CfgConsistency.ql reports 0 violations on a full CPython database.

Copilot

Pull request overview

Migrates Python’s “new” dataflow library to use the shared CFG facade (semmle.python.controlflow.internal.Cfg) and the new shared-SSA-backed adapter, updating framework models, query code, and reblessing test expectations to match the new CFG node canonicalization/toString behavior.

Changes:

Switch Python dataflow/type-tracking/guard plumbing to shared CFG + shared SSA adapter (with corresponding API updates across libraries and queries).
Update framework models and security/query libraries to reference Cfg:: nodes and new SSA entities.
Rebless a large set of library/query-test .expected files for new node labels (e.g. After ...) and known semantic deltas.

Show a summary per file

File	Description
python/tools/recorded-call-graph-metrics/ql/lib/RecordedCalls.qll	Adjust call-graph metrics plumbing for CFG changes (note: current diff introduces a removed API reference; see PR comment).
python/ql/test/query-tests/Statements/exit/UseOfExit.expected	Reblessed expected output for new CFG node stringification.
python/ql/test/query-tests/Security/CWE-942-CorsMisconfigurationMiddleware/CorsMisconfigurationMiddleware.expected	Reblessed expected results with `After ...` node labels.
python/ql/test/query-tests/Security/CWE-798-HardcodedCredentials/HardcodedCredentials.expected	Reblessed path-graph labels (legacy `ControlFlowNode for ...` → new labels).
python/ql/test/query-tests/Security/CWE-732-WeakFilePermissions/WeakFilePermissions.expected	Reblessed expected results for new CFG node representation.
python/ql/test/query-tests/Security/CWE-614-InsecureCookie/InsecureCookie.expected	Reblessed expected results for new CFG node representation.
python/ql/test/query-tests/Security/CWE-327-InsecureDefaultProtocol/InsecureDefaultProtocol.expected	Reblessed expected results for new CFG node representation.
python/ql/test/query-tests/Security/CWE-327-BrokenCryptoAlgorithm/BrokenCryptoAlgorithm.expected	Reblessed expected results for new CFG node representation.
python/ql/test/query-tests/Security/CWE-326-WeakCryptoKey/WeakCryptoKey.expected	Reblessed expected results for new CFG node representation.
python/ql/test/query-tests/Security/CWE-312-CleartextStorage-py3/CleartextStorage.expected	Reblessed dataflow path output for new CFG labels.
python/ql/test/query-tests/Security/CWE-295-RequestWithoutValidation/RequestWithoutValidation.expected	Reblessed expected results for new CFG node representation.
python/ql/test/query-tests/Security/CWE-295-MissingHostKeyValidation/MissingHostKeyValidation.expected	Reblessed expected results for new CFG node representation.
python/ql/test/query-tests/Security/CWE-215-FlaskDebug/FlaskDebug.expected	Reblessed expected results for new CFG node representation.
python/ql/test/query-tests/Security/CWE-209-StackTraceExposure/ExceptionInfo.expected	Updates expectations to record known “missing result” regressions under new CFG exception modeling.
python/ql/test/query-tests/Security/CWE-1275-SameSiteNoneCookie/SameSiteNoneCookie.expected	Reblessed expected results for new CFG node representation.
python/ql/test/query-tests/Security/CWE-113-HeaderInjection/Tests2-with-wsgi-validator/HeaderWriteTest.expected	Reblessed taint-tracking output labels for new CFG nodes.
python/ql/test/query-tests/Security/CWE-1004-NonHttpOnlyCookie/NonHttpOnlyCookie.expected	Reblessed expected results for new CFG node representation.
python/ql/test/query-tests/Security/CWE-089-SqlInjection/CONSISTENCY/DataFlowConsistency.expected	Reblessed consistency-test expected results for new CFG labels.
python/ql/test/query-tests/Security/CWE-089-SqlInjection-local-threat-model/SqlInjection.expected	Reblessed path-graph labels for new CFG nodes.
python/ql/test/query-tests/Security/CWE-079-Jinja2WithoutEscaping/Jinja2WithoutEscaping.expected	Reblessed expected results for new CFG node representation.
python/ql/test/query-tests/Security/CWE-074-TemplateInjection/TemplateInjection.expected	Reblessed expected results for new CFG node representation.
python/ql/test/query-tests/Security/CWE-020-IncompleteHostnameRegExp/IncompleteHostnameRegExp.expected	Reblessed expected output labels.
python/ql/test/query-tests/Numerics/Pythagorean.expected	Reblessed expected output labels.
python/ql/test/query-tests/Imports/deprecated/DeprecatedModule.expected	Reblessed and records an additional expected deprecated-module result.
python/ql/test/query-tests/Functions/ModificationOfParameterWithDefault/test.expected	Updates expectations to record missing inline-annotation results.
python/ql/test/query-tests/Expressions/super/CallToSuperWrongClass.expected	Reblessed expected output labels.
python/ql/test/query-tests/Exceptions/general/IncorrectExceptOrder.expected	Updates expectations to record known missing alert under new exception reachability.
python/ql/test/query-tests/Exceptions/general/EmptyExcept.expected	Reblessed expected results; includes additional expected findings.
python/ql/test/query-tests/Exceptions/general/CatchingBaseException.expected	Reblessed expected results; removes one expected finding.
python/ql/test/query-tests/Classes/subclass-shadowing/SubclassShadowing.expected	Reblessed expected output labels.
python/ql/test/query-tests/Classes/multiple/multiple-init/SuperclassInitCalledMultipleTimes.expected	Reblessed expected output labels.
python/ql/test/query-tests/Classes/multiple/multiple-del/SuperclassDelCalledMultipleTimes.expected	Reblessed expected output labels.
python/ql/test/query-tests/Classes/init-calls-subclass-method/InitCallsSubclassMethod.expected	Reblessed expected output labels.
python/ql/test/library-tests/PointsTo/new/ImpliesDataflow.ql	Updates points-to/dataflow bridge logic for new CFG/legacy interop.
python/ql/test/library-tests/PointsTo/new/ImpliesDataflow.expected	Reblessed expected output labels.
python/ql/test/library-tests/frameworks/stdlib/InlineTaintTest.expected	Updates expectations to record missing taint annotations under new CFG behavior.
python/ql/test/library-tests/frameworks/stdlib-py2/ConceptsTest.expected	Updates expectations to record missing concept annotation.
python/ql/test/library-tests/frameworks/sqlalchemy/ConceptsTest.expected	Updates expectations to record missing/spurious concept results.
python/ql/test/library-tests/frameworks/rest_framework/CONSISTENCY/DataFlowConsistency.expected	Reblessed expected output labels.
python/ql/test/library-tests/frameworks/modeling-example/SharedCode.qll	Updates example framework model to use `Cfg::` node types.
python/ql/test/library-tests/frameworks/modeling-example/ProperModel.ql	Updates example model steps to use `Cfg::` node types.
python/ql/test/library-tests/frameworks/modeling-example/NaiveModel.ql	Updates example model steps to use `Cfg::` node types.
python/ql/test/library-tests/frameworks/lxml/InlineTaintTest.expected	Updates expectations to record missing taint annotations under new CFG behavior.
python/ql/test/library-tests/frameworks/django/CONSISTENCY/DataFlowConsistency.expected	Reblessed expected output labels.
python/ql/test/library-tests/frameworks/django-orm/NormalDataflowTest.expected	Updates expectations to record an unexpected flow result under new CFG/SSA.
python/ql/test/library-tests/frameworks/cryptography/EcKeygenOrigin.expected	Reblessed expected output labels.
python/ql/test/library-tests/frameworks/aiohttp/InlineTaintTest.ql	Updates guard predicate signatures and uses `Cfg::` call/name nodes.
python/ql/test/library-tests/frameworks/aiohttp/InlineTaintTest.expected	Updates expectations to record missing taint annotations under new CFG behavior.
python/ql/test/library-tests/essa/ssa-compute/CONSISTENCY/TypeTrackingConsistency.expected	Removes expected unreachable-node violations (rebless under new behavior).
python/ql/test/library-tests/dataflow/use-use-flow/use-use-counts.ql	Switches SSA entities to `SsaImpl::...` and CFG node types to `Cfg::...`.
python/ql/test/library-tests/dataflow/use-use-flow/use-use-counts.expected	Reblessed expected output, including implicit-use behavior changes.
python/ql/test/library-tests/dataflow/typetracking/tracked.ql	Updates AST↔CFG mapping usage to avoid removed `getAFlowNode` patterns.
python/ql/test/library-tests/dataflow/typetracking/test.py	Updates inline annotations (marking tracked/spurious) to match new behavior.
python/ql/test/library-tests/dataflow/typetracking/moduleattr.expected	Reblessed expected output labels and entry-definition naming.
python/ql/test/library-tests/dataflow/typetracking-summaries/tracked.ql	Updates tracked-node selection to use `Cfg::NameNode`.
python/ql/test/library-tests/dataflow/typetracking_imports/highlight_problem.ql	Updates SSA variable/type usage and normal-exit matching under new CFG.
python/ql/test/library-tests/dataflow/typetracking_imports/highlight_problem.expected	Reblessed expected output labels for SSA defs.
python/ql/test/library-tests/dataflow/tainttracking/TestTaintLib.qll	Updates taint-test config to use `Cfg::` call/name nodes.
python/ql/test/library-tests/dataflow/tainttracking/customSanitizer/test.py	Updates inline expectations to mark known missing taint results under exception reachability changes.
python/ql/test/library-tests/dataflow/tainttracking/customSanitizer/InlineTaintTest.ql	Updates guard predicates to use `Cfg::ControlFlowNode` and `Cfg::CallNode`.
python/ql/test/library-tests/dataflow/tainttracking/customSanitizer/InlineTaintTest.expected	Reblessed expected sanitizer-node labels.
python/ql/test/library-tests/dataflow/tainttracking/basic/LocalTaintStep.expected	Reblessed expected output labels.
python/ql/test/library-tests/dataflow/tainttracking/basic/GlobalTaintTracking.ql	Updates taint config to use `Cfg::` call/name nodes.
python/ql/test/library-tests/dataflow/tainttracking/basic/GlobalTaintTracking.expected	Reblessed expected output labels.
python/ql/test/library-tests/dataflow/summaries/TestSummaries.qll	Updates summary matching to use `Cfg::NameNode` for call identification.
python/ql/test/library-tests/dataflow/strange-essaflow/testFlow.ql	Switches ESSA entities to `SsaImpl::...` for import-flow test logic.
python/ql/test/library-tests/dataflow/strange-essaflow/testFlow.expected	Reblessed expected output labels.
python/ql/test/library-tests/dataflow/regression/custom_dataflow.ql	Updates regression config to use `Cfg::` call/name nodes.
python/ql/test/library-tests/dataflow/regression/custom_dataflow.expected	Reblessed expected output and records additional flow.
python/ql/test/library-tests/dataflow/module-initialization/localFlow.ql	Updates module-initialization flow test to use `SsaImpl::...` definitions.
python/ql/test/library-tests/dataflow/method-calls/test.expected	Reblessed expected output labels.
python/ql/test/library-tests/dataflow/global-flow/test.py	Updates inline expectations to mark spurious write under new SSA pruning/behavior.
python/ql/test/library-tests/dataflow/fieldflow/UnresolvedCalls.ql	Updates unresolved-call expectations to use `Cfg::CallNode`.
python/ql/test/library-tests/dataflow/fieldflow/UnresolvedCalls.expected	Reblessed expected output labels.
python/ql/test/library-tests/dataflow/def-use-flow/def_use_counts.ql	Switches SSA entities to `SsaImpl::...` and CFG node types to `Cfg::...`.
python/ql/test/library-tests/dataflow/coverage/test.py	Updates inline expectation to record a missing flow result.
python/ql/test/library-tests/dataflow/coverage/localFlow.expected	Reblessed expected output labels and entry-definition naming.
python/ql/test/library-tests/dataflow/coverage/argumentRoutingTest.ql	Updates routing test to use `Cfg::` node types and `SsaImpl::...` definitions.
python/ql/test/library-tests/dataflow/basic/sources.expected	Reblessed expected output labels (including pre/post nodes).
python/ql/test/library-tests/dataflow/basic/sinks.expected	Reblessed expected output labels (including pre/post nodes).
python/ql/test/library-tests/dataflow/basic/maximalFlows.expected	Reblessed expected output labels.
python/ql/test/library-tests/dataflow/basic/localStep.expected	Reblessed expected output labels.
python/ql/test/library-tests/dataflow/basic/callGraphSources.expected	Reblessed expected output labels.
python/ql/test/library-tests/dataflow/basic/callGraphSinks.expected	Reblessed expected output labels.
python/ql/test/library-tests/dataflow/basic/callGraph.expected	Reblessed expected output labels.
python/ql/test/library-tests/ApiGraphs/py3/test_crosstalk.expected	Reblessed expected output labels.
python/ql/test/library-tests/ApiGraphs/py3/ModuleImportWithDots.expected	Reblessed expected output labels.
python/ql/test/experimental/query-tests/Security/CWE-611-SimpleXmlRpcServer/SimpleXmlRpcServer.expected	Reblessed expected output labels.
python/ql/test/experimental/query-tests/Security/CWE-347/JWTMissingSecretOrPublicKeyVerification.expected	Reblessed expected output labels.
python/ql/test/experimental/query-tests/Security/CWE-347/JWTEmptyKeyOrAlgorithm.expected	Reblessed expected output labels.
python/ql/test/experimental/query-tests/Security/CWE-346/CorsBypass.expected	Reblessed expected output labels.
python/ql/test/experimental/query-tests/Security/CWE-338/InsecureRandomness.expected	Reblessed expected output labels.
python/ql/test/experimental/query-tests/Security/CWE-287/ImproperLdapAuth.expected	Reblessed expected output labels.
python/ql/test/experimental/query-tests/Security/CWE-208/TimingAttackAgainstSensitiveInfo/PossibleTimingAttackAgainstSensitiveInfo.expected	Reblessed expected output labels.
python/ql/test/experimental/query-tests/Security/CWE-208/TimingAttackAgainstHeaderValue/TimingAttackAgainstHeaderValue.expected	Reblessed expected output labels.
python/ql/test/experimental/query-tests/Security/CWE-208/TimingAttackAgainstHash/TimingAttackAgainstHash.expected	Reblessed expected output labels.
python/ql/test/experimental/query-tests/Security/CWE-208/TimingAttackAgainstHash/PossibleTimingAttackAgainstHash.expected	Reblessed expected output labels.
python/ql/test/experimental/query-tests/Security/CWE-094/Js2Py.expected	Reblessed expected output labels.
python/ql/test/experimental/meta/InlineTaintTest.qll	Updates inline taint-test harness to use `Cfg::NameNode`/`Cfg::CallNode`.
python/ql/test/experimental/library-tests/FindSubclass/Find.expected	Updates expected results (removing one subclass member).
python/ql/test/experimental/library-tests/CallGraph/InlineCallGraphTest.ql	Updates call-graph comparison tests to bridge legacy points-to calls with new CFG wrappers.
python/ql/test/experimental/library-tests/CallGraph-type-annotations/InlineCallGraphTest.expected	Updates expectations for missing/fixed call-graph edges under new resolution behavior.
python/ql/test/experimental/import-resolution/importflow.ql	Updates import-resolution test logic to use `Cfg::CompareNode`/`Cfg::ControlFlowNode`.
python/ql/test/experimental/import-resolution-namespace-relative/test.ql	Updates taint config to use `Cfg::NameNode` in call identification.
python/ql/test/experimental/attrs/AttrWrites.expected	Reblessed expected output labels.
python/ql/test/experimental/attrs/AttrReads.expected	Reblessed expected output labels.
python/ql/test/2/query-tests/Expressions/UseofInput.expected	Reblessed expected output labels.
python/ql/test/2/query-tests/Expressions/UseofApply.expected	Reblessed expected output labels.
python/ql/test/2/query-tests/Exceptions/raising/RaisingTuple.expected	Reblessed expected output labels.
python/ql/test/2/query-tests/Exceptions/generators/UnguardedNextInGenerator.expected	Updates expected results (adds additional expected findings).
python/ql/src/Statements/UseOfExit.ql	Updates query to use `Cfg::CallNode` from shared CFG facade.
python/ql/src/Statements/SideEffectInAssert.ql	Updates CFG node typing to `Cfg::ControlFlowNode`.
python/ql/src/Statements/ModificationOfLocals.ql	Updates query logic to use shared CFG facade node types.
python/ql/src/Security/CWE-798/HardcodedCredentials.ql	Updates query to use shared CFG facade node types in matching.
python/ql/src/Security/CWE-327/Ssl.qll	Updates modeling to use `Cfg::AttrNode` in augmented-assignment patterns.
python/ql/src/Security/CWE-327/PyOpenSSL.qll	Updates attribute-node usage to `Cfg::AttrNode`.
python/ql/src/Security/CWE-079/Jinja2WithoutEscaping.ql	Updates call AST access via `Cfg::CallNode` wrappers.
python/ql/src/Security/CWE-020-ExternalAPIs/ExternalAPIs.qll	Updates resolved-call predicate to accept `Cfg::CallNode`.
python/ql/src/Resources/FileNotAlwaysClosedQuery.qll	Updates BasicBlock typing to shared `Cfg::BasicBlock` facade.
python/ql/src/meta/analysis-quality/TTCallGraphShared.ql	Updates analysis-quality query to use `Cfg::CallNode`.
python/ql/src/meta/analysis-quality/TTCallGraphOverview.ql	Updates analysis-quality aggregation to use `Cfg::CallNode`.
python/ql/src/meta/analysis-quality/TTCallGraphNewAmbiguous.ql	Updates analysis-quality query to use `Cfg::CallNode`.
python/ql/src/meta/analysis-quality/TTCallGraphNew.ql	Updates analysis-quality query to use `Cfg::CallNode`.
python/ql/src/meta/analysis-quality/TTCallGraphMissing.ql	Updates analysis-quality query to use `Cfg::CallNode`.
python/ql/src/meta/analysis-quality/TTCallGraph.ql	Updates analysis-quality query to use `Cfg::CallNode`.
python/ql/src/meta/analysis-quality/CallGraphQuality.qll	Updates call-graph quality plumbing to use `Cfg::CallNode`.
python/ql/src/Functions/SignatureOverriddenMethod.ql	Updates call resolution to use `Cfg::CallNode` in DataFlow call nodes.
python/ql/src/Expressions/UseofApply.ql	Updates query to use `Cfg::CallNode`.
python/ql/src/experimental/semmle/python/security/injection/CsvInjection.qll	Updates guard predicate signature to use `Cfg::ControlFlowNode`.
python/ql/src/experimental/Security/UnsafeUnpackQuery.qll	Updates CFG node type checks to `Cfg::...` nodes.
python/ql/src/experimental/Security/CWE-770/UnicodeDoS.ql	Updates compare-node typing and guard signature to `Cfg::...` nodes.
python/ql/src/experimental/Security/CWE-346/CorsBypass.ql	Removes legacy Flow import and rewrites to shared CFG facade node types.
python/ql/src/experimental/Security/CWE-340/TokenBuiltFromUUID.ql	Updates definition-node typing to `Cfg::DefinitionNode`/`Cfg::NameNode`.
python/ql/src/experimental/Security/CWE-022bis/TarSlipImprov.ql	Updates CFG node type checks to `Cfg::...` nodes.
python/ql/src/Exceptions/UnguardedNextInGenerator.ql	Mixes shared CFG facade + legacy Flow for call matching under generator/exception behavior.
python/ql/lib/utils/test/dataflow/UnresolvedCalls.qll	Updates unresolved-call test utility to use `Cfg::CallNode` and canonical representative filtering.
python/ql/lib/utils/test/dataflow/testTaintConfig.qll	Updates taint test config to use `Cfg::...` node types.
python/ql/lib/utils/test/dataflow/testConfig.qll	Updates dataflow test config to use `Cfg::NameNode`.
python/ql/lib/utils/test/dataflow/RoutingTest.qll	Updates routing-test helpers to use `Cfg::CallNode` in name extraction.
python/ql/lib/utils/test/dataflow/NormalTaintTrackingTest.qll	Updates sink reconstruction to use `Cfg::NameNode`.
python/ql/lib/utils/test/dataflow/NormalDataflowTest.qll	Updates sink reconstruction to use `Cfg::NameNode`.
python/ql/lib/utils/test/dataflow/MaximalFlowTest.qll	Updates maximal-flow config to use `Cfg::...` node types.
python/ql/lib/semmle/python/security/dataflow/UrlRedirectCustomizations.qll	Updates binary-expr node typing to `Cfg::BinaryExprNode`.
python/ql/lib/semmle/python/security/dataflow/TarSlipCustomizations.qll	Updates guard signature and attribute/name traversal to `Cfg::...` nodes.
python/ql/lib/semmle/python/security/dataflow/ServerSideRequestForgeryCustomizations.qll	Updates binary-expr typing and guard signature to `Cfg::...` nodes.
python/ql/lib/semmle/python/security/dataflow/ExceptionInfo.qll	Updates caught-exception modeling to use `Cfg::NameNode.defines` instead of legacy ESSA node definitions.
python/ql/lib/semmle/python/regexp/internal/ParseRegExp.qll	Updates binary-expr typing to `Cfg::BinaryExprNode`.
python/ql/lib/semmle/python/frameworks/Yarl.qll	Updates guard signature to use `Cfg::ControlFlowNode`.
python/ql/lib/semmle/python/frameworks/Yaml.qll	Updates framework call node override type to `Cfg::CallNode`.
python/ql/lib/semmle/python/frameworks/Werkzeug.qll	Updates subscript/definition typing to `Cfg::SubscriptNode`/`Cfg::DefinitionNode`.
python/ql/lib/semmle/python/frameworks/Twisted.qll	Rewrites return-value flow node selection using AST `Return` rather than legacy helper.
python/ql/lib/semmle/python/frameworks/Tornado.qll	Updates multiple node types to `Cfg::...` and adjusts routing helpers to new node classes.
python/ql/lib/semmle/python/frameworks/Stdlib/Urllib.qll	Updates guard signature to use `Cfg::ControlFlowNode`.
python/ql/lib/semmle/python/frameworks/Pyramid.qll	Rewrites return-value flow node selection using AST `Return` rather than legacy helper.
python/ql/lib/semmle/python/frameworks/Pydantic.qll	Updates subscript node typing to `Cfg::SubscriptNode`.
python/ql/lib/semmle/python/frameworks/Pycurl.qll	Updates attribute node typing to `Cfg::AttrNode`.
python/ql/lib/semmle/python/frameworks/MarkupSafe.qll	Updates call/binary-expr override node types to `Cfg::...`.
python/ql/lib/semmle/python/frameworks/internal/SubclassFinder.qll	Updates CFG node typing in subclass-finding logic to `Cfg::ControlFlowNode`.
python/ql/lib/semmle/python/frameworks/Gradio.qll	Updates list-node typing to `Cfg::ListNode`.
python/ql/lib/semmle/python/frameworks/FastApi.qll	Rewrites return-value node selection using AST `Return`; updates subscript/definition typing to `Cfg::...`.
python/ql/lib/semmle/python/frameworks/Bottle.qll	Rewrites return-value node selection using AST `Return`; updates subscript/definition typing to `Cfg::...`.
python/ql/lib/semmle/python/Flow.qll	Minor comment text changes (some appear accidental; see PR comments).
python/ql/lib/semmle/python/dataflow/new/SensitiveDataSources.qll	Updates sensitive-data modeling to use `Cfg::...` node types.
python/ql/lib/semmle/python/dataflow/new/internal/VariableCapture.qll	Updates variable-capture integration to use shared CFG+SSA adapter types.
python/ql/lib/semmle/python/dataflow/new/internal/TypeTrackingImpl.qll	Updates summary return-node matching and node-type checks to shared CFG/SSA types.
python/ql/lib/semmle/python/dataflow/new/internal/TaintTrackingPrivate.qll	Updates binary/subscript/call node typing and with-definition SSA references.
python/ql/lib/semmle/python/dataflow/new/internal/MatchUnpacking.qll	Updates pattern-alias/capture flow to use CFG-based name nodes.
python/ql/lib/semmle/python/dataflow/new/internal/LocalSources.qll	Updates subscript node typing to `Cfg::SubscriptNode`.
python/ql/lib/semmle/python/dataflow/new/internal/ImportStar.qll	Updates name node typing and import-star base to `Cfg::...` nodes.
python/ql/lib/semmle/python/dataflow/new/internal/Builtins.qll	Updates builtin-access discovery to use `Cfg::NameNode`.
python/ql/lib/semmle/python/dataflow/new/BarrierGuards.qll	Updates guard signatures and iterable/compare node typing to `Cfg::...`.
python/ql/lib/semmle/python/Concepts.qll	Updates guard signatures to use `Cfg::ControlFlowNode`.
python/ql/lib/semmle/python/ApiGraphs.qll	Imports shared CFG facade and updates subscript/definition node typing; includes comment text tweaks (some need correction; see PR comments).
python/ql/consistency-queries/DataFlowConsistency.ql	Updates dataflow consistency check to use `Cfg::CallNode` for call matching.

Copilot's findings

Files reviewed: 204/249 changed files
Comments generated: 8

 ResolvableRecordedCall() {
- exists(Call call, XmlCallee xmlCallee, ControlFlowNode callCfg |
+ exists(Call call, XmlCallee xmlCallee |
 call = this.getACall() and
- callCfg.getNode() = call and
- calleeValue.getACall() = callCfg and
+ calleeValue.getACall() = call.getAFlowNode() and
 xmlCallee = this.getXmlCallee() and


 predicate strictlyDominates(ControlFlowNode other) {
 // This predicate is gigantic, so it must be inlined.
- // About 1.4 billion tuples for OpenStack Cinder.
+ // About 1.4 billion tuples for OpenStack Py::Cinder.
 this.getBasicBlock().strictlyDominates(other.getBasicBlock())


 * Holds if this element is at the specified location.
 * The location spans column `startcolumn` of line `startline` to
 * column `endcolumn` of line `endline` in file `filepath`.
- * For more information, see
+ * Py::For more information, see
 * [Locations](https://codeql.github.com/docs/writing-codeql-queries/providing-locations-in-codeql-queries/).


 ResolvableRecordedCall() {
- exists(Call call, XmlCallee xmlCallee, ControlFlowNode callCfg |
+ exists(Call call, XmlCallee xmlCallee |
 call = this.getACall() and
- callCfg.getNode() = call and
- calleeValue.getACall() = callCfg and
+ calleeValue.getACall() = call.getAFlowNode() and
 xmlCallee = this.getXmlCallee() and


 predicate strictlyDominates(ControlFlowNode other) {
 // This predicate is gigantic, so it must be inlined.
- // About 1.4 billion tuples for OpenStack Cinder.
+ // About 1.4 billion tuples for OpenStack Py::Cinder.
 this.getBasicBlock().strictlyDominates(other.getBasicBlock())


 * Holds if this element is at the specified location.
 * The location spans column `startcolumn` of line `startline` to
 * column `endcolumn` of line `endline` in file `filepath`.
- * For more information, see
+ * Py::For more information, see
 * [Locations](https://codeql.github.com/docs/writing-codeql-queries/providing-locations-in-codeql-queries/).


+// Importing python under the `py` namespace to avoid importing `Cfg::CallNode` from `Flow.qll` and thereby having a naming conflict with `API::CallNode`.
 private import python as PY
+private import semmle.python.controlflow.internal.Cfg as Cfg


 // TODO: once convenient, this should be done at a higher level than the AST,
 // at least at the CFG layer, to take splitting into account.
- // Also consider `SequenceNode for generality.
+ // Also consider `Cfg::SequenceNode for generality.
 exists(PY::List list | list = pred.(DataFlow::ExprNode).getNode().getNode() |


yoff · 2026-06-01T12:49:34Z

Addressed in latest push (force-pushed amendment):

RecordedCalls.qll:245 — restored P1's ControlFlowNode callCfg | callCfg.getNode() = call form. (Already addressed in the prior amendment; this was an accidental regression because the flip's tree was taken from the older big PR before P1 landed.)
Flow.qll:202 (OpenStack Py::Cinder) — reverted to OpenStack Cinder. P2 already had this fix in its review-comment amendment; this regression came from the same older-tree path. Flip should not touch Flow.qll at all and now doesn't.
Flow.qll:1098 (Py::For more information) — same as above; reverted to For more information.
ApiGraphs.qll:9 — reworded comment to clarify that the conflict is with the unqualified CallNode from import python (via Flow.qll), not Cfg::CallNode.
ApiGraphs.qll:779 — closed the unclosed backtick: \Cfg::SequenceNode` for generality`.

yoff · 2026-06-01T13:20:19Z

Re. PatternAliasDefinition / PatternCaptureDefinition replacement in MatchUnpacking.qll:

Precision analysis. The legacy code used PatternAliasDefinition pad (an EssaNodeDefinition produced by pattern_alias_definition(v, defn) in SsaDefinitions.qll), constrained as defn.getNode() = alias ∧ alias = v.getAStore(). The v.getAStore() filter is vacuous here — MatchAsPattern.getAlias() always returns a binding-position Name, so it is always a store. So pad.getDefiningNode() is just "the CFG node for the alias Name".

In the new CFG, a leaf Name AST has exactly one canonical Cfg::NameNode. So matching by getNode() = alias picks the same single CFG node the legacy ESSA wrapper picked. No precision is lost.

Simplification. Pushed 8178a8d to inline the intermediate exists(Cfg::ControlFlowNode aliasCfg | ...) to nodeTo.(CfgNode).getNode().getNode() = alias (and analogous for capture.getVariable()), and removed the now-unused Cfg import. library-tests/dataflow/match still passes.

yoff · 2026-07-02T09:28:15Z

Performance fix in reaction to the DCA run

The DCA run on this commit surfaced large analysis-time regressions on several projects (e.g. ICTU__quality-time 54s → 9,286s, biosimulations__biosimulations 45s → 3,843s, ytdl-org__youtube-dl 121s → 521s).

Root cause. The regression was almost entirely in API::Node.getSubscriptAt/1 (ApiGraphs.qll) — ~74% of total predicate time in the youtube-dl evaluator log. Two compounding factors:

Under the new SSA/CFG, type-tracking reaches ~5× more subscript expressions, inflating API subscript edges (~112k → ~548k on youtube-dl).
The QL join order (chosen by the optimizer from the shifted cardinality statistics) materialised a 3.8-billion-tuple intermediate — getObject() = this.getAValueReachableFromSource() was joined before the much more selective getIndex()/result-edge constraints.

Fix (result-preserving). The subscripting branch of getSubscriptAt now binds subscript from the already-pinned result edge (result.asSource()/asSink()) first, then getIndex(), and only then performs the large getObject() = …getAValueReachableFromSource() join — so that join becomes a membership check on a bound node rather than a cross-product. This changes only the join order; the result set is identical.

Validation (cold cache, youtube-dl, identical 538,621 result pairs before and after):

database	before	after
worst-case DB	8m41s	19.9s (~26×)
controlled A/B DB	2m15s	20.6s (~6.5×)

The ApiGraphs library tests and the NoSqlInjection query tests all pass with no result drift.

Note on #22101. That PR is orthogonal to this regression: it rewrites the dict/zip/enumerateflow summaries (targeting a different youtube-dl bottleneck from #21888), and does not touch the API-graph / type-tracking / getSubscriptAt path.

Follow-up. The residual ~5× subscript-edge inflation stems from a systemic non-canonicalization in the CFG facade (typed node classes match every before/after/canonical CFG variant of an AST node, and their child accessors are dominance-based and therefore non-functional). A systematic cleanup — introducing a CanonicalControlFlowNode (this.injects(_)) and using it as the result type of the facade's typed node classes and child accessors — is planned as a separate follow-up after this stack merges. The join-order fix above already recovers ~95% of the lost time on its own.

yoff · 2026-07-02T15:15:54Z

Second performance fix from the DCA follow-up (separate commit `c2f439a`)

The re-run of DCA after the getSubscriptAt join-order fix showed that a few projects had recovered (e.g. ytdl-org__youtube-dl), but two of the worst regressions were essentially unchanged:

source	before	after
`ICTU__quality-time`	58s	9,316s (~160×)
`biosimulations__biosimulations`	48s	3,622s (~75×)

Root cause (a distinct, second hotspot). Reproducing ICTU/quality-time locally, the suite stalled for 20+ minutes inside a single predicate — Cfg::ControlFlowNode.strictlyDominates/1.

The Cfg::ControlFlowNode facade re-exports the shared CFG library's dominates/strictlyDominates, which are declared bindingset[this, that] + pragma[inline_late] (i.e. intended as bound-pair membership checks). The facade wrappers dropped those annotations (plain pragma[inline]). The only callers — the with / async with taint steps in DataFlowPrivate.qll and TaintTrackingPrivate.qll — do bind both endpoints, but without the bindingset the optimizer was free to materialise the full strictlyDominates relation (O(nodes²) over the larger shared-CFG node set), which is what blew up.

Fix (result-preserving). Restore bindingset[this, other] + pragma[inline_late] on the two facade wrappers. Only the binding annotations change; the predicate body is untouched, so results are identical.

Validation.

ICTU/quality-time, full python-security-extended suite: >20 min stall → ~6 min (cold cache).
ControlFlow + dataflow/coverage library tests: all 59 pass.

Why a separate commit. This is deliberately added as a new commit on top of the branch rather than folded into the commit that introduced the facade (#21921). Keeping the perf fix in its own commit — and not rewriting history now that the stack is under review — preserves the existing review state, CI results, and DCA runs.

Flips the Python dataflow trunk from the legacy CFG (semmle/python/Flow.qll) and legacy ESSA SSA (semmle/python/essa/*) to the new shared CFG facade (semmle.python.controlflow.internal.Cfg) and the new SSA adapter (semmle.python.dataflow.new.internal.SsaImpl), both introduced additively in the preceding PRs in this stack. This is the trunk-flip equivalent of the original draft PR #21894 (kept around as documentation), rebased on top of the four preparatory PRs: P1: Remove AstNode.getAFlowNode() and rewrite callers (#21919). P2: Qualify Flow.qll's AST references with Py:: prefix (#21920). P3: Add new shared-CFG-backed control flow graph (#21921). P4: Add new shared-SSA-backed SSA adapter (#21923). The Python dataflow library (semmle/python/dataflow/new/) now imports the new CFG facade and SSA adapter. All CFG-typed predicates (ControlFlowNode, CallNode, BasicBlock, NameNode, AttrNode, ...) are qualified with the Cfg:: prefix; SSA references switch from EssaVariable/EssaDefinition to SsaImpl::Definition/SourceVariable. GuardNode is redesigned to use the new CFG's outcome-node model (isAfterTrue / isAfterFalse) instead of the legacy ConditionBlock + flipped indirection. Only BarrierGuard<...> is preserved as public API. Framework files (Bottle, FastApi, Django, Tornado, Pyramid, Stdlib, ...) are updated to take CFG nodes from the new facade. A handful of dataflow consistency tweaks for the new CFG: - Augmented-assignment targets are treated as both load and store. - 'from X import *' produces uncertain SSA writes for unknown names. - CFG nodes are canonicalised so dataflow does not see equivalent pre/post-order pairs as distinct nodes. Two AST tweaks for the new CFG: - AstNodeImpl: omit PEP 695 type-parameter names from FunctionDefExpr / ClassDefExpr children. - ImportResolution: drop the legacy essa import. Test churn (~175 files): reblessed library- and query-test .expected files reflect slightly different CFG granularity, different toString output, and a handful of true alert deltas in security queries. Verification: all 367 lib + src + consistency-queries compile clean. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The `Cfg::ControlFlowNode` facade re-exports the shared CFG library's `dominates`/`strictlyDominates` predicates, which are declared `bindingset[this, that]` + `pragma[inline_late]` and are meant to be used as bound-pair membership checks. The facade wrappers dropped these annotations (using plain `pragma[inline]`), so even though the only callers — the `with` / `async with` taint steps in DataFlowPrivate.qll and TaintTrackingPrivate.qll — bind both endpoints, the optimizer was free to materialise `Cfg::ControlFlowNode.strictlyDominates/1` as a full O(nodes^2) relation over the (larger) shared-CFG node set. On some projects this dominated analysis time entirely (DCA showed e.g. ICTU/quality-time and biosimulations regressing ~75-160x). Restoring `bindingset[this, other]` + `pragma[inline_late]` on the wrappers turns the predicate back into a bound-pair check and is result-preserving (only binding annotations change, the predicate body is unchanged). Reproduced on ICTU/quality-time: full python-security-extended suite went from stalling >20min on `strictlyDominates` to completing in ~6min; all ControlFlow and dataflow/coverage library tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

CopilotAI review requested due to automatic review settings June 1, 2026 12:29

yoff requested a review from a team as a code owner June 1, 2026 12:29

Copilotstarted reviewing on behalf of yoff June 1, 2026 12:29 View session

github-actionsBot added the Python label Jun 1, 2026

yoff mentioned this pull request Jun 1, 2026
Yoff/python use shared cfg for dataflow #21894
Draft

CopilotAI reviewed Jun 1, 2026
View reviewed changes

yoff force-pushed the yoff/python-shared-cfg-dataflow-flip branch 2 times, most recently from 4304ddb to 5187e9aCompare June 1, 2026 12:49

yoff marked this pull request as draft June 1, 2026 13:15

yoff force-pushed the yoff/python-shared-cfg-dataflow-flip branch from 5187e9a to 8178a8dCompare June 1, 2026 13:18

yoff force-pushed the yoff/python-add-new-ssa branch from 28062e1 to 4d4b916Compare June 1, 2026 13:28

yoff force-pushed the yoff/python-shared-cfg-dataflow-flip branch from 8178a8d to 8ecea67Compare June 1, 2026 13:37

yoff mentioned this pull request Jun 1, 2026
Python: simplify decorator-detection predicates to pure AST match #21926
Merged

yoff force-pushed the yoff/python-add-new-ssa branch from 4d4b916 to 6313e70Compare June 1, 2026 14:05

yoff force-pushed the yoff/python-shared-cfg-dataflow-flip branch from 8ecea67 to c476e5aCompare June 1, 2026 15:46

yoff mentioned this pull request Jun 2, 2026
Python: inline init_module_submodule_defn into ImportResolution #21930
Merged

yoff force-pushed the yoff/python-add-new-ssa branch from 6313e70 to e3b9611Compare June 2, 2026 08:30

yoff force-pushed the yoff/python-shared-cfg-dataflow-flip branch from c476e5a to 483a64aCompare June 2, 2026 08:30

github-actionsBot added the documentation label Jun 2, 2026

yoff force-pushed the yoff/python-add-new-ssa branch from e3b9611 to 9be5c62Compare June 2, 2026 08:47

yoff force-pushed the yoff/python-shared-cfg-dataflow-flip branch from 483a64a to ead93caCompare June 2, 2026 08:47

github-actionsBot removed the documentation label Jun 2, 2026

yoff force-pushed the yoff/python-add-new-ssa branch from 9be5c62 to 944d56eCompare June 2, 2026 13:48

yoff force-pushed the yoff/python-shared-cfg-dataflow-flip branch from ead93ca to 4f4119fCompare June 2, 2026 13:48

yoff force-pushed the yoff/python-add-new-ssa branch from 944d56e to a9015e4Compare June 2, 2026 13:56

yoff force-pushed the yoff/python-shared-cfg-dataflow-flip branch 2 times, most recently from 01b9e1f to 45aaf8dCompare June 2, 2026 14:02

yoff force-pushed the yoff/python-add-new-ssa branch from a9015e4 to 91b812eCompare June 2, 2026 14:02

yoff force-pushed the yoff/python-add-new-ssa branch from 948382f to b7f79f2Compare June 22, 2026 13:47

yoff force-pushed the yoff/python-shared-cfg-dataflow-flip branch from fa159d2 to 408ba62Compare June 22, 2026 13:47

yoff force-pushed the yoff/python-add-new-ssa branch from b7f79f2 to bebe8c9Compare June 24, 2026 08:04

yoff force-pushed the yoff/python-shared-cfg-dataflow-flip branch from 408ba62 to 93cae5fCompare June 24, 2026 08:04

yoff force-pushed the yoff/python-add-new-ssa branch from bebe8c9 to b3f1d94Compare June 24, 2026 08:18

yoff force-pushed the yoff/python-shared-cfg-dataflow-flip branch from 93cae5f to acb4a58Compare June 24, 2026 08:18

yoff force-pushed the yoff/python-add-new-ssa branch from b3f1d94 to e51acdcCompare June 24, 2026 08:46

yoff force-pushed the yoff/python-shared-cfg-dataflow-flip branch from acb4a58 to 5081d81Compare June 24, 2026 08:46

yoff force-pushed the yoff/python-add-new-ssa branch from e51acdc to 0389e01Compare June 25, 2026 22:20

yoff force-pushed the yoff/python-shared-cfg-dataflow-flip branch 2 times, most recently from 4cbe7ae to 62f34d5Compare June 25, 2026 23:08

yoff force-pushed the yoff/python-add-new-ssa branch from 0389e01 to 6c6af73Compare June 25, 2026 23:08

yoff force-pushed the yoff/python-shared-cfg-dataflow-flip branch from 62f34d5 to 44b8aadCompare June 29, 2026 11:40

github-actionsBot added C# JS C++ Java Go Ruby QL-for-QL Rust Pull requests that update Rust code Swift Kotlin DataFlow Library Actions Analysis of GitHub Actions labels Jun 29, 2026

yoff mentioned this pull request Jul 2, 2026
Python: unpin legacy CFG/ESSA from the AST cached stage (DCA measurement) #22116
Draft

yoffand others added 2 commits July 30, 2026 15:30

Navigation Menu

Uh oh!

Python: switch dataflow library to new (shared) CFG + SSA - #21925

Python: switch dataflow library to new (shared) CFG + SSA#21925
yoff wants to merge 2 commits into
yoff/python-add-new-ssafrom
yoff/python-shared-cfg-dataflow-flip

yoff commented Jun 1, 2026•
edited
Loading

Uh oh!

CopilotAI left a comment

Uh oh!

yoff commented Jun 1, 2026

Uh oh!

yoff commented Jun 1, 2026

Uh oh!

yoff commented Jul 2, 2026

Uh oh!

yoff commented Jul 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

yoff commented Jun 1, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

What changes

Dataflow library

GuardNode redesign

Framework updates

Dataflow consistency tweaks for the new CFG

Two AST tweaks for the new CFG

Test churn (~135 reblessed .expected)

Alert preservation — investigation

Snapshot-freshness fix

Verification

Uh oh!

CopilotAI left a comment

Choose a reason for hiding this comment

Pull request overview

Copilot's findings

Uh oh!

yoff commented Jun 1, 2026

Uh oh!

yoff commented Jun 1, 2026

Uh oh!

yoff commented Jul 2, 2026

Performance fix in reaction to the DCA run

Uh oh!

yoff commented Jul 2, 2026

Second performance fix from the DCA follow-up (separate commit c2f439a)

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

yoff commented Jun 1, 2026•
edited
Loading

Test churn (~135 reblessed `.expected`)

Second performance fix from the DCA follow-up (separate commit `c2f439a`)