task: build SemanticIndex and the concept-retrieval pipeline

[serina-mcfall]

Summary

Implements all 10 steps of #210's plan: SemanticIndex and the concept-retrieval pipeline
(concept → subsystem → candidate symbols → confirmed references) from the design doc's Data
Model item 3 and Concept Retrieval reasoning rules -- including both a positive worked example
(a genuinely vague question resolving to a real symbol) and the negative flow-tracing boundary
case, both reproduced live against this repo's own real code.

Stacked on #213/#214 (this branch is task/207-project-graph + this PR's own commits) since
#210 has a real, stated dependency on #206 for content to embed/summarize and calls into #207's
ProjectGraph for confirmation. Base is task/207-project-graph, not launchpad -- review this
PR's own commits (semantic_index.py, test_semantic_index.py, the plan doc); the earlier
commits are #213/#214's, already under review there.

Related issue

Refs #210 (a stacked PR's base isn't the default branch, so GitHub won't create a real closing
link from "Closes" here -- retarget to launchpad once #213/#214 merge, then this can close #210
for real)

Issue type

Task

---

Agent provenance

Field	Value
Harness / provider	Claude Code
Model	claude-sonnet-5
Session reference	N/A - harness does not expose a session/run URL
Initiating human	@serina-mcfall

Objective

Build SemanticIndex and the concept-retrieval pipeline, per the design doc's Data Model item 3
and Concept Retrieval reasoning rules.

Impacted components

launchpad/project-intelligence/semantic_index.py
launchpad/project-intelligence/test_semantic_index.py

Approach and rejected alternatives

Chosen: "subsystem" is a real second ConceptEntry level scoped to file (the schema's own stated
scope kinds are symbol_id | file | doc_section), with a genuine two-stage search() -- rank
files first, then rank symbols within the top file(s). Rejected collapsing subsystem and symbol
into one flat ranking, since the issue's own pipeline shape names both stages explicitly.

Chosen: per-symbol ConceptEntry.scope is symbol_id, not qualified_name. Found empirically
(not assumed) while live-verifying against all 453 real buzz-core symbols: multiple distinct
symbols share one short qualified_name (e.g. several build_event functions in different
modules), so qualified_name is not a safe unique key -- symbol_id (a real per-symbol-unique
RepoQL URI from #206's index_crate()) is, and the design doc names it as a valid scope kind
too. Added qualified_name_for() to translate back, since #207's ProjectGraph addresses nodes
by qualified_name, not symbol_id.

Chosen: embeddings are a bag-of-words frequency vector over an identifier-aware tokenizer
(camelCase/snake_case splitting), not a trained ML embedding model -- matches #210's own "out of
scope: any embedding-model selection process beyond what's needed to demonstrate the pipeline
once." A symbol's own identity (kind + qualified_name + signature) is weighted 2x over context
mentions (calls/tests/config/docs) in its embedding (embed_symbol()) -- found empirically
necessary while verifying the positive worked example: without it, a caller's summary absorbs
its callees' name tokens too (since "calls X" contributes X's own identifier tokens), so the
caller (is_unshared_gated_event) initially outranked the callee it calls
(is_shared_gated_kind) itself for a query about the callee's own behavior.

Chosen: confirm_via_graph() returns empty tuples (never hidden) when nothing confirms a
candidate, so a caller can see an unconfirmed guess for what it is rather than a confirmation
that looks the same as a confirmed one.

Flagged in the plan under OPEN (not decided silently): the bag-of-words embedding choice itself,
top_k defaults (3 at each stage), and whether summary generation should ever call an LLM rather
than a deterministic template (it doesn't, consistent with every other module this session).

Verification

Command run:

$ python3 -m unittest test_semantic_index

Raw output:

...........................
----------------------------------------------------------------------
Ran 27 tests in 0.003s

OK

Command run (final CLI -- both worked examples, live against the real buzz-core index):

$ python3 semantic_index.py

Raw output:

=== Positive worked example: concept -> subsystem -> candidate -> confirmation ===

Concept: 'which function decides if a kind is gated for shared visibility'
  Subsystem: crates/buzz-core/src/kind.rs (score=0.4184)
  Candidate: is_shared_gated_kind (score=0.5706)
    confirmed called_by: tests::shared_gated_kinds_membership
    confirmed called_by: is_unshared_gated_event
    confirmed tested_by: tests::shared_gated_kinds_membership

=== Negative worked example: flow-tracing boundary ===

Concept posed to THIS pipeline (which it should NOT be used for): 'what does the author-always-allowed test call, two calls deep'
  This pipeline resolved to: 'tests::test_unspecified'
  One-hop confirmation targets: (none)
  is_shared_gated_kind (the real 2-hop answer) never appears above: True

  Contrast -- ProjectGraph.reachable() answers the IDENTICAL relationship exactly:
    verified path: tests::is_unshared_gated_event_author_always_allowed -> is_unshared_gated_event -> is_shared_gated_kind

• Tests or checks were run and the raw output is pasted above
• The diff is confined to the scope of the linked issue
• No secrets, keys, tokens or hostnames were added to tracked files

Not verified

Did not test the pipeline against any crate other than buzz-core, or against docs (the
doc_section scope kind is supported by the schema but nothing in this PR populates one -- no
doc-indexing pipeline exists yet). Did not tune the bag-of-words/identity-weighting approach
against a broad corpus of concept queries -- identity_weight=2.0 was checked empirically against
this PR's own two worked examples only, not derived analytically or validated at scale; a human
should treat retrieval quality as unproven beyond these two cases. Did not test behavior when the
RepoQL host is down (host was confirmed healthy via host status before every live verification
in this PR).

Security implications

None -- pure in-process data structures and string/math operations, no I/O beyond the existing
indexer.build_index()/graph.ProjectGraph calls this task ingests from (already reviewed under
#213/#214).

Escalations

Whether the bag-of-words/identity-weighting embedding approach is an acceptable long-term
retrieval strategy, or purely a placeholder to satisfy #210's own "demonstrate the pipeline once"
scope -- flagged in the plan's OPEN section and again here. This PR implements exactly what
#210's Definition of done requires and no more; deciding whether real embeddings are needed later
is a call for a human, not something to silently upgrade or silently leave unstated.

[benmitchell11]

Confirmed the stacking is legitimate — diff against task/207-project-graph only touches this PR's own 3 files (plan doc, semantic_index.py, test_semantic_index.py), nothing re-touching #213/#214.

The symbol_id-not-qualified_name scope key is a genuinely good empirical catch — finding that multiple real symbols in this codebase share a short qualified_name (several build_event functions in different modules) by checking against all 453 actual indexed symbols, rather than assuming qualified_name was unique, is exactly the kind of thing that would otherwise surface as a confusing bug much later. qualified_name_for() and its own test confirm the translation path back to what ProjectGraph expects is real, not just described.

The negative worked example is the more valuable of the two demonstrations here — showing this pipeline correctly resolves a flow-tracing question to the wrong answer (one hop, not the real two-hop relationship) and explicitly contrasting it against ProjectGraph.reachable() giving the right one, is a genuine boundary-case proof rather than only a positive demo. confirm_via_graph() returning empty rather than something that looks like a confirmation when nothing actually confirms a candidate is the right failure shape for a retrieval tool a human or agent might trust.

Good escalation on the embedding approach — bag-of-words with an empirically-tuned identity weight is correctly scoped as satisfying #210's 'demonstrate once' requirement, not silently presented as a validated retrieval strategy.

[tucktuck101]

Recommendation: approve — but hold the merge until #214 lands and this retargets to launchpad (per the PR description's own note; #214 currently has changes requested, so this stays queued behind it).

This cleanly fulfills #210's Definition of done, and I verified rather than trusted: I ran the suite against the base branch's graph.py/symbol.py — 27/27 pass — and CI's executed checks are green (Rust/e2e jobs correctly path-skipped).

What stands out positively:

• The DoD's two hard requirements are genuinely proven, not narrated. The positive worked example asserts the exact resolved symbol and its confirmation edges, with a test guaranteeing the concept sentence contains no substring of the target name — so it can't be an accidental keyword hit. The negative flow-tracing case is the stronger demonstration: it shows this pipeline structurally cannot express a 2-hop answer (no hop/path fields exist, asserted via dataclasses.fields), then contrasts reachable() verifying the identical relationship exactly.
• confirm_via_graph() really calls #207's edges_from() — structural confirmation, not similarity re-dressed — and returns empty tuples rather than something that masquerades as a confirmation.
• symbol_id-keyed scopes with qualified_name_for() translation is a real empirical catch (colliding qualified_names in live buzz-core), with its own regression test.
• Cold-start (find_it_for_me on an empty index) and zero-vector cosine are both handled and tested; the identity_weight=2.0 constant is explained rather than left magic.

Non-blocking observations (fine to defer; none needs to hold up the eventual merge):

1. No minimum-score floor. An entirely unrelated query (I tried "zebra xylophone") still yields a top candidate at candidate_score=0.0, complete with real confirmation edges. The scores are exposed on PipelineResult, so callers can filter — but consider either a small threshold or a docstring warning in find_it_for_me() that a 0.0-score candidate is noise, since the confirmation attached to it can make it look legitimate.
2. Tie-break nondeterminism. search() iterates self._file_scopes (a set) before sorting; for tied scores the winner depends on set iteration order, which varies across processes under hash randomization. Sorting ties by scope name would make results reproducible.
3. Final ranking ignores subsystem_score. search() sorts the flattened results by candidate_score alone, so a lexically strong symbol in a weakly matched file (within the top-k files) can outrank a candidate from the best-matched subsystem — and it can return up to top_k² rows. Documented behavior and the plan flags top_k semantics as unspecified, so just noting it as a known shape.
4. Inherited from #207: ProjectGraph merges symbols sharing a qualified_name onto one node, so confirm_via_graph() on a colliding name conflates both symbols' edges. This PR correctly works around it on the index side; worth a tracking note on the graph side.

The bag-of-words embedding stand-in is correctly scoped against #210's explicit out-of-scope line and flagged in the plan's OPEN section for human confirmation — the right escalation rather than a silent product decision.