Epic: durable versioned library publishing — storage abstraction for go get / pip / npm

[antoinetoussaint-byte]

Epic: durable, versioned library publishing — a storage abstraction consumable via native package managers (go get, pip, npm)

Sibling of the CLI-owned-build epic (#443): same durability principle, applied to libraries instead of service images. A consumer must be able to pull a codefly library at a pinned version without the codefly toolchain or local source — using the language's native package manager.

Backend for now: GitHub (matches how modules use git tags and agents use GitHub releases). The store is an abstraction so PyPI/Artifactory/npm-registry/Go-proxy can drop in later.

---

The gap (why this exists)

Library is already a versioned resource (library.codefly.yaml: semver Version, Languages[] exports for go/python/typescript, Git config) resolved from git tags (resources.LibraryResolver.ResolveVersion). But consumption is local-dev-only: SetupLocalDevelopment wires Go replace directives, pip install -e, and npm link against local source paths. codefly install library and codefly list libraries are stubs. There is no library agent RPC and no library proto service.

Net: a service can use a library only if the library's source is present locally — the exact shape of the #443 builder/ problem. There is no durable, versioned, toolchain-free consumption path.

---

Core idea: a backend-abstract library store that maps library@version → native package coordinates

Key realization: GitHub is natively a Go module registry (repo + semver tag = go get). For Python/npm it also works (Releases + PEP 503 index / GitHub Packages). One GitHub-backed store serves all three via per-language strategies; the interface stays backend-agnostic.

// pkg/librarystore
type Language string // "go" | "python" | "typescript"

type Coordinates struct { Language Language; Name string; Version string } // semver

type Published struct {
 Coordinates
 ImportPath string // go module path / pip dist name / @scope/name
 Ref string // IMMUTABLE anchor: git commit or release id
 Location string // URL
 Digest string // sha256 of the published artifact
 InstallHint string // "go get X@vN" / "pip install X==N --extra-index-url …" / "npm i @org/X@N"
}

type Store interface {
 Publish(ctx context.Context, artifactDir string, c Coordinates) (Published, error)
 Resolve(ctx context.Context, language Language, name, constraint string) (Published, error) // semver → concrete
 List(ctx context.Context, language Language, name string) ([]string, error)
}

This is the "storage abstraction with version" the epic is about. It reuses the immutability trick already in the module base-source lock (record repo/ref/commit + content digest; verify the tag never moved).

---

Per-language mapping (honest about the hard parts)

• Go — the clean one. To be go get-able the module path must equal its git URL. Recommend: one published repo per Go export (github.com/<org>/<lib>-go, tagged vX.Y.Z) → go get + GOPROXY caching just work, no toolchain. (Alternative: monorepo subtree with lib/vX.Y.Z tag prefixes — works but couples module paths to the monorepo layout.)
• Python — no clean native "versioned git-subdir with deps" install. Two tiers:
  • MVP: direct ref pip install "name @ git+https://github.com/<org>/<repo>@vX.Y.Z" (works today, GitHub-only, no index).
  • Proper: agent builds wheel+sdist → store uploads as GitHub Release assets → store maintains a PEP 503 "simple" index as static files on a GitHub Pages branch → pip install --extra-index-url https://<org>.github.io/<idx>/simple name==X.Y.Z.
• npm / TS — GitHub Packages npm registry (native, scoped @org/name), or MVP npm i github:org/repo#semver:^X.

---

Publish flow (mirrors the #443 recipe: package → verify → publish → record + SBOM)

codefly export library <name>:

1. Resolve / bump the version.
2. Per LanguageExport: the library agent packages that export into a destination (Deploy-pattern output_directory + returned inventory/digest — a new small library-packaging RPC, symmetric to the build-recipe plan). MVP can skip the agent and publish the source subtree directly.
3. Verify the packaged tree against the inventory (same handshake as VerifyDockerBuildPlan).
4. store.Publish(...) per language → Published coordinates.
5. Record library-export.codefly.json (versions, coordinates, digests) + emit the library SBOM (reuse the Builder.SBOM wiring added in Generated builder/ Dockerfiles are not durable — consumers can't rebuild images without the agent toolchain #443) as a release asset — durability + provenance parity with images.

---

Consumer flow (finally implement install library)

Extend resources.LibraryResolver with a published mode beside the existing local-dev mode: store.Resolve(name, constraint) → native coordinates wired into go.mod (require), pyproject/requirements (+ index), package.json (+ registry). codefly sync library-dependencies picks local-dev (replace/-e/link) or published (go get/pip/npm) per flag. codefly install library / list libraries stop being stubs.

---

Phases (same dependency order as #443)

• P0 — MVP, GitHub-only, no agent-contract change. pkg/librarystore interface + GitHub Go strategy (repo+tag → native go get), Python/npm via git/direct refs; codefly export library; resolver published-mode; implement install library. Delivers durable, toolchain-free consumption fast.
• P1 — proper indexes + agent packaging. Python PEP 503 index on GitHub Pages + wheels in Releases; npm via GitHub Packages; library-packaging RPC + verify + SBOM.
• P2 — pluggable backends + integrity. Non-GitHub stores (PyPI/Artifactory/npm registry/Go proxy); immutability locks; signed provenance.

---

Decisions (proposed defaults — override if you disagree)

1. Go layout: ✅ one published repo per library (cleanest go get) — vs monorepo subtree + tag-prefix.
2. Python: ✅ start MVP git+https direct refs → then PEP 503 GitHub-Pages index in P1.
3. First cut: ✅ P0 MVP.
4. Surface: ✅ new codefly export library (avoids colliding with codefly publish used for agent/module releases); install library for the consumer side.

---

Open questions

• Where do published Go repos live — same org (codefly-dev) or a per-workspace/product org? Auto-create with gh repo create, or pre-provisioned?
• Private vs public published repos (affects go get auth: GOPRIVATE, ~/.netrc, GONOSUMCHECK)?
• Do libraries need cross-language version lockstep (one library version → all language exports tagged together), or independent per-language versions?

First PR (in progress): pkg/librarystore — the Store interface + GitHub Go strategy (publish repo+tag, resolve/list via git tags, go get install hint), tested end-to-end against a local bare git repo.

[antoinetoussaint-byte]

P0 started: #447 lands the pkg/librarystore foundation — the backend-agnostic Store + the GitHub Go strategy (publish repo+tag → native go get, immutable versions, semver resolve), tested end-to-end against a real bare git repo. Next in P0: codefly export library command + resolver published-mode (install library).