[refactor] Investigate core's CGO dependency surface and where the CGO-free switch should live

[antoinetoussaint-byte]

Context

cli#456 hit a concrete instance of a general problem: codefly companion publish and codefly self build --os/--arch cross-compile the CLI with CGO_ENABLED=0 (statically linked for alpine), but the CLI's in-process source behavior installed core's tree-sitter semantic analyzer (core/code/semantic, WithSemanticAnalyzer(semantic.New())), which requires cgo. CGO_ENABLED=0 go build failed to link (build constraints exclude all Go files for the tree-sitter bindings).

The cli-side fix (cli#456) splits newSource on an explicit codefly_nosemantic build tag: the default variant keeps the analyzer (and still fails loudly under an accidental CGO_ENABLED=0 build), the tagged variant falls back to core's CGO-free NewDefaultCodeServer. The build scripts pass -tags codefly_nosemantic.

That works, but it pushes core's cgo boundary knowledge into a downstream consumer. This issue is to investigate the CGO surface generically in core, where the cgo code and the expertise about it live.

Motivation: cgo creep is a known pain

This is not hypothetical. Core has already experienced cgo-dependency creep — notably through the plugin/language-server surface — and untangling it after the fact was painful. cgo dependencies are insidious because they are transitive and silent: a new plugin or a bumped dependency can pull a cgo-only package into the import graph, and nothing fails until someone tries a CGO_ENABLED=0 build far downstream (a companion publish, an alpine image), long after the offending change merged. The fix is then a cross-repo archaeology exercise. The goal here is to make the CGO-free surface an enforced contract in core so creep fails at the core PR that introduces it, not at a downstream publish weeks later.

Problem statement

cgo-ness is a property of the import graph, resolved at link time. Any binary that transitively imports a cgo-only package (today: core/code/semantic → tree-sitter Go bindings) cannot be built CGO-free, regardless of how cleanly the package is separated. Package isolation is necessary but not sufficient to make a consumer CGO-free — the consumer still needs a build-time switch to conditionally import (or not import) the isolated package.

Core already did the package isolation well: core/code/semantic is a separate package; NewDefaultCodeServer is CGO-free and never imports it. What core does not provide is (a) a supported, build-tag-aware way to say "default server, plus semantics when the build allows it" — so every consumer reinvents the cgo/no-cgo split, and (b) any guardrail that stops a new cgo dependency from silently re-entering the CGO-free surface.

Investigation scope

1. Inventory the cgo surface in core. Which packages require cgo, directly or transitively (tree-sitter grammars are the known one — confirm there are no others: bindings, sqlite, plugin/language-server code, etc.). Document each with what pulls it in, and which are reachable from the surface that is supposed to be CGO-free.
2. Decide the ownership boundary for the cgo/no-cgo switch. Options, roughly in increasing cost:
   • (a) Leave it in consumers. Fine while the CLI is the only one; revisit on a second consumer.
   • (b) Own the split in core. Expose a build-tag-split constructor (e.g. code.NewDefaultCodeServerWithSemantics(root) — cgo file installs the analyzer, !cgo/tagged file returns the plain default) so consumers get one call and no build tags of their own. Makes codefly_nosemantic (or a core-canonical tag name) a public core contract; consumers still pass -tags on CGO-free builds. Standardize the tag name here.
   • (c) Move cgo out-of-process. Serve tree-sitter semantics from a separate cgo helper/agent so core-linked consumers can be fully CGO-free with no split anywhere. Matches where per-service semantics already live (language agents over gRPC). Largest change; adds a process/IPC hop; contradicts the in-process source-only path. Evaluate whether it's where the gateway is heading regardless.
3. Pick and document the canonical build-tag name and its semantics (loud-fail-on-accidental-CGO_ENABLED=0 is a desirable property — preserve it under whatever design is chosen).
4. Guardrail against creep (the core deliverable). A CI check in core that builds the intended-CGO-free surface with CGO_ENABLED=0 and asserts it links — so a plugin or dependency that drags cgo back into that surface fails in core, at the PR that introduces it, with a clear message, instead of surfacing at a downstream companion publish. cli#456 added exactly this guard on the cli side; core needs the authoritative analogue. Consider also a documented allowlist of "packages permitted to require cgo" so additions are a conscious, reviewed decision.

Non-goals

• Not proposing to drop the semantic analyzer from the normal CLI/gateway — it's part of that contract and stays cgo-linked there.
• Not blocking cli#456, which is a correct, self-contained fix; this issue is the generalization it revealed.

Reference

• cli#456 — fix: build a CGO-free CLI so companion publish links statically (the concrete instance + the cli-side split and CI guard).

[antoinetoussaint-byte]

Concrete downstream instance of this cgo surface: #346 — the service agents require cgo (each links its own tree-sitter grammar), which is exactly why none of them ships a linux/arm64 build today. If the agents were CGO-free (an outcome this investigation could recommend), #346 would collapse to a trivial goarch: arm64 matrix add instead of needing the aarch64 cross-cgo toolchain.