Skip to content

Builtins get two can:// identities: PyCG spells the module <builtin>, Jedi spells it builtins #132

Description

@rahlk

Problem

The same builtin function gets two distinct can:// identities, because Jedi and PyCG spell
the builtins module differently and nothing normalizes it before ids are minted.

Measured on test/fixtures/whole_applications/requests at -a 2 (main @ 6f02581):

external_symbols under @external/<builtin>/ : 29
external_symbols under @external/builtins/ : 14
names present under BOTH spellings : 12
all, any, chr, getattr, hasattr, isinstance, iter, len, max, print, setattr, sorted
call_graph edges -> @external/<builtin>/... : 130 all prov=['pycg']
call_graph edges -> @external/builtins/... : 79 all prov=['jedi']

The provenance split is exact: <builtin> is PyCG's spelling, builtins is Jedi's.
core.py:_home_external_symbols derives the id by splitting the dotted signature on its last
dot (module, name = sig.rsplit(".", 1)) and never canonicalizes module, so both spellings
mint their own @external home.

Two consequences:

  1. A consumer asking "who calls len" gets two disjoint answers, and neither is complete.
  2. Provenance can never merge for builtins.merge_edges coalesces on (src, dst); because
    the two backends produce different dst ids for the same target, a call both resolvers agree
    on can never reach prov: ["jedi", "pycg"]. In the same run, 198 non-builtin edges do carry
    both — so the merge works, and builtins are structurally excluded from it.

PyCG only ever emits the bare <builtin> module (69 occurrences across requests + flask);
every dotted form (builtins.str, builtins.dict, …) comes from Jedi. So an exact-match alias
is sufficient — no prefix rewriting needed.

Scope boundary

Canonicalizes PyCG's module spelling so one symbol has one id. Does not change how either
resolver resolves calls, does not touch the @external id grammar, does not address Jedi's
resolution failures on attribute calls (a capability limit, not a bug), and does not change
body.callee backfill — which was verified correct: across all 234 unresolved call sites in
requests, every one has callee_signature is None, and there are zero cases where a resolved
signature failed to reach its body node.

Goals

  • Canonicalize <builtin>builtins on PyCG edge endpoints, at the single exit point
    (PyCG.build_call_graph_edges) so every shard strategy is covered
  • Re-coalesce PyCG's own edges after canonicalization, so a normalized duplicate sums its
    weight instead of emitting two edges with identical endpoints
  • Verify prov: ["jedi", "pycg"] now appears for builtins both resolvers agree on
  • No @external/<builtin>/ id remains in any emitted payload

Caveats and known risks

  • Normalization must happen before merge_edges, not at id-minting time.core.py:598
    merges Jedi and PyCG edges on raw dotted signatures, and ids are minted later at line 633.
    Canonicalizing at the minting site would leave two already-merged edges with identical
    endpoints and split provenance — the symptom would move rather than disappear.
  • Edge weights change. Coalescing two spellings into one sums their weights, so any test
    pinning a builtin edge weight will need updating. That is the correct new value, not a
    regression.
  • _coalesce_edges (pycg_analysis.py:468) is broken and sits on this path (reachable from
    line 710): it constructs PyCallEdge(source=…, target=…) and reads existing.source, but the
    model's fields are src/dst with no aliases — so it raises whenever two shards produce the
    same (src, dst). This fix must not route through it. Tracked separately.
  • <builtin> is PyCG's convention and could change if PyCG is ever replaced; the alias map
    should be named for what it is (a backend spelling quirk), not treated as a general grammar.

Definition of done

  • @external/<builtin>/ appears zero times in analysis.json for requests, flask, xarray
  • The 12 doubly-spelled names collapse to one id each, with weights summed
  • At least one builtin edge carries prov: ["jedi", "pycg"] where previously none could
  • A regression test asserts a hand-written expected id for a builtin call resolved by both backends
  • Existing call-graph tests pass, with any weight changes updated deliberately

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions