Skip to content

Schema v2 migration + four analysis levels (epic) #69

Description

@rahlk

Epic: migrate codeanalyzer-python to canonical schema v2 + four analysis levels

Migrate the analyzer from the v1 payload (PyApplication + a separate program_graphs section) to the CLDK canonical schema v2 — one additive Code Property Graph tree — exposed as four gated analysis levels (-a 1|2|3|4), in both the analysis.json and Neo4j projections. This is a breaking major release (the codeanalyzer-backend "existing analyzer → schema v2" path). It supersedes the L3 approach in #67: dataflow moves inline into the tree, and the interprocedural SDG moves from -a 3 to -a 4.

Target shape

application → symbol_table{module} → types/functions → callables{ body{}, cfg[], cdg[], ddg[], summary[] }
call_graph[], param_in[], param_out[] — cross-function edges, at application scope
  • can://python/<app>/<file>/<type>/<callable-sig> durable ids (≥ callable); <callable-id>@<line>:<col> / @tag ordinal ids (< callable).
  • source stored once per module + byte-offset spans (every node's text slices off it); the per-callable code field is dropped.
  • Payload envelope: schema_version: "2.0.0", language, max_level, k_limit.

Levels

LevelGrowsAdds edges
1tree to callable + call nodes in body (callee:null)
2callee: null→id backfillcall_graph
3rest of body + @entry/@exitcfg, cdg, ddg (syntactic, prov:["ssa"])
4synthetic param verticesparam_in, param_out, summary, + ddg (semantic, prov:["points-to"])

--graphs sdg requires -a 4 (requesting it at -a 3 is a flag error).

Key decisions

  • Reuse the existing dataflow kernels (cfg/dominance/defuse/sdg/summaries/alias); a new identity-mapping layer translates internal (signature, node_id) keys → can:// ids at emit time, which keeps the JSON and Neo4j projections in lockstep.
  • Scalpel (python-scalpel) is the primary L4 points-to oracle behind the frozen may_alias interface, with TypeBasedAliasOracle as an automatic fallback.
  • Analyzer-only scope. The python-sdk Pydantic revision is a separate follow-up (via cldk-sdk-frontend). Taint stays deferred to the SDK; the backward slicer stays internal as the L3/L4 validation gate (no product "queryable slicer" framing, no taint_flows section).
  • Neo4j is a re-identify + one prov property + source/span parity + MAJOR bump (schema.neo4j.json2.0.0); the near-identity projection structure is unchanged. Always full-depth.

Invariants / gates

  • Additive superset: analysis.json(-a 1) ⊆ … ⊆ analysis.json(-a 4), modulo the callee: null→id refinement and the L3→L4 DDG ssapoints-to widening.
  • Two-projection agreement: JSON node/edge ids == Neo4j merge-key values.
  • Per-level gates (validated per stage): L1 canonical-schema conformance; L2 no dangling call-graph endpoints; L3 intraprocedural backward slice == hand-computed set; L4 no dangling SDG endpoints + PARAM arity + a summary for a known transitive flow.

Stages

Each stage is an independently shippable PR-unit with its own gate. Child issues:

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions