gen-pipe — scoped channels + dataflow algebra

⚠ ARCHIVED — this library has retired. Bind gen-view, and gen-select for sel.

ADR-0010 §3 retires gen-pipe into the movement vocabulary, and that ruling gained a fourth destination on 2026-08-20: gen-view, the substrate's derived-view constructor. Twelve of the seventeen exports are retired into named gen-view constructs; sel retires into gen-select, which consumers should bind directly. No new consumer should bind any of them.

The retirement completed on 2026-08-21. gen-pipe is off the gen hub's mkGenLibs roster — binding and stratum line dropped together — and is no longer a hub flake input. The repository is orphaned for reference under ADR-0031 F3: no content is deleted, nothing here is maintained, and it gains no new consumers.

The surface is MARKED, not removed. ADR-0031 F3 — "no content is deleted" — applied one level down, at export granularity. Every export still evaluates and the suite is unchanged.

Three things are HELD rather than retired, because nothing is dropped silently and no home exists yet: contribute (ruled to a producer declaration that no document defines), deferred (assigned to the eval-boundary crossing by a spec the landed crossing does not carry), and consume's class-discipline half (ruled to the delivery-class realization surface, which is not yet extracted). consume's type-check half is gen-types' by ruling, with a stated shape divergence. One further export, join, has no single destination construct: its fan-in survives as a caller-side composition, recorded as expressible rather than homed.

Per-export destinations: REFERENCE.md · AGENTS.md · lib/default.nix.

The content-agnostic dataflow algebra for scoped channels. A channel is a typed, named accumulation lane; its value at a scope position is a deterministic fold over the contributions visible from that position under a pinned traversal. Operators (map, filter, fold, scan, over, route, join, tee) connect channels into a dataflow DAG, validated at composition time and evaluated demand-driven.

gen-pipe supplies three guarantees that den v1's pipe machinery enforced only by convention:

1. Determinism as law — B5 per channel: pinned traversal (self → imports → parent), associative-only combine, no silent reorder or dedup.
2. Provenance as data — every contribution carries its producer as structured registry-entry identities (entity, scope coordinates, aspect); operators extend the chain; consumers can interrogate it.
3. Class as type — contributions are class-tagged at emission; a deferred (config-demanding) contribution's config means the producing class's config at the producing scope; cross-class consumption requires a declared adapter; class ambiguity is a definition-time error naming the producer and channel.

Plus a fourth, for the class-share build path: config-dependence as a static flag — every contribution and derived output carries classInvariant, derived from arg-shape and composed through operators at composition time (never a runtime discovery).

Table of Contents

• Overview
• Gen Ecosystem
• Quick Start
• The dual-inclusion question
• API Reference
• Testing
• Theoretical Foundations

Overview

A channel's value at a position p is:

sequence(ch, p) = base(ch, p) ++ delivered(e₁, p) ++ … ++ delivered(eₙ, p)
value(ch, p)    = foldl combine init (postDedup (sequence ch p))

• base is the channel's native sequence — for a declared channel, traversal.contributionsAt enumerated in the traversal pin (self → imports → parent, imports in declaration order — gen-scope's collectionAttr public contract). For a derived channel, the defining operator's output.
• e₁ … eₙ are the inbound route/tee delivery edges, in compose declaration order, each an order-preserving select-matched subsequence of its source at the same position.
• postDedup applies the channel's declared, traced dedup policy (identity if none). combine is associative-only; the default is list concatenation, so the default channel value is the ordered contribution list.

gen-pipe does not evaluate NixOS/home-manager configs, discover writers, or construct scope graphs — those reach it through a caller-supplied traversal adapter (den-hoag's projection of the scope graph). It reads the graph and produces plain data; no output can feed graph structure (the two-stratum discipline).

Gen Ecosystem

Library	Role
gen-prelude	Pure nixpkgs-lib-free utility base (builtins re-exports + vendored lib utils)
gen-graph	The ordering front door — compose validates its channel DAG with topoOrder
gen-select	Selector algebra (pattern matching over graph positions) — routing/consumption predicates
gen-scope	HOAG scope-graph evaluator; supplies the collectionAttr traversal contract gen-pipe consumes
gen-pipe	This lib — scoped channels + dataflow algebra

Class B: L1-only dependencies, nothing upward, nixpkgs-lib-free. gen-schema is not a dependency — registry entries are opaque id_hash-bearing values to gen-pipe (duck-typed).

Quick Start

let
  genPipe = import (builtins.fetchGit "https://github.com/sini/gen-pipe") { };
  inherit (genPipe) channel contribute compose run sel;

  # 1. declare a channel
  backends = channel { name = "http-backends"; };

  # 2. build contributions (den-hoag does this per producing position)
  c = contribute {
    channel = backends;
    value = [ "10.0.0.1" ];
    producer = { entity = hostEntry; scope = { host = hostEntry; }; aspect = aspectEntry; };
  };

  # 3. compose the DAG
  dag = compose [ backends ];

  # 4. run against a traversal adapter (gen-scope's collectionAttr shape)
  outputs = run {
    inherit dag;
    traversal = {
      order = p: [ p ];                              # self → imports → parent
      contributionsAt = p: chName: [ c ];
      classesOf = p: [ ];
      render = coords: "…";
    };
  };
in
  (outputs.at position)."http-backends".values     # ⇒ [ "10.0.0.1" ]

The dual-inclusion question

An aspect A contributes to channel C. A is included by BOTH user sini and host axon-01. What class is the contribution? What does config mean inside it? How does it fail?

Dual inclusion never yields one ambiguous contribution — it yields two, one per (aspect, producing position) pair, each class-tagged by its own producing position. The user-position contribution is tagged home-manager; the host-position one nixos. Inside a deferred value, config means the producing class's config at the producing scope (never the consuming module's) — which is what makes cross-scope consumption sound. The identity key includes the producing scope coordinates, so the pair is never a dedup duplicate.

It fails loudly, never silently: an ambiguous producing class (0 or ≥2 classes, no explicit tag) is E1; cross-class consumption without an adapter is E2; a bare config-demanding function without the deferred wrapper is E8. See REFERENCE.md §Class semantics.

API Reference

Full API in REFERENCE.md. Surface:

genPipe = {
  channel  = { name, type ?, merge ?, combine ?, init ?, dedup ?, class ? }: <channel>;
  contribute = { channel, value, producer, class ? }: <contribution>;
  deferred = fn: <deferredValue>;

  map    = f: <channel>: <channel>;           # per-element; or map { f, name ?, site ?, type ?, dedup ?, class ? }
  filter = p: <channel>: <channel>;
  fold   = { f, init, ... }: <channel>: <channel>;
  scan   = { f, init, ... }: <channel>: <channel>;
  over   = f: <channel>: <channel>;           # WHOLE-list f : [a] -> [b] (value-demanding); or over { f, ... }
  route  = { from, select, to }: <op>;
  join   = { inputs, combine ?, site ?, ... }: <channel>;
  tee    = { from, outputs }: <op>;

  compose = decls: <dag>;
  run     = { dag, traversal }: <outputs>;
  consume = { outputs, at, channel, class ?, adapters ?, select ?, mode ? }: [ value ] | [ record ];

  provenanceOf = record: <chain>;
  traceOf      = { outputs, at, channel }: <trace>;
  sel = <gen-select's library surface>;       # constructors (entity/kind/star/…) + adapters + matches
};

All public entry points pass and receive values — channel values, contribution records, registry entries — never "kind:name" strings (the identity law). Strings appear only as internal keys and in rendered error/display text.

Testing

cd ci && nix-unit --flake .#tests            # or: nix flake check

Every law (L1–L13) is a named test group; error content is golden-tested; the goldens re-run against a real gen-scope graph in integration-gen-scope.

Theoretical Foundations

• Kahn (1974), KPN — informed-by only. gen-pipe channels have multiple writers, violating Kahn's single-writer condition (the source of KPN determinism). gen-pipe's determinism therefore does not come from KPN semantics — it comes from the B5 discipline: pinned canonical traversal + associative-only left fold. This caveat is load-bearing. ★ And the positive half of that argument cites a document this repository does not carry. "HOAG r2 §B5, den ISSUES #10" is an external section reference; git ls-files lists no spec document, as AGENTS.md § What "B5" denotes records. The content of B5 is not lost — this repository restates it in its own words as law L1 in REFERENCE.md § Laws (left fold in the pinned canonical traversal under an associative-only combine, no silent reorder, no silent dedup), and that restatement, not the unheld section, is what the code and its suites are checkable against. So read the coordinate as provenance, unverifiable here, in the sense gen-graph marks Cited unverified. Whether B5 should instead be re-grounded in-repo as its own stated law, or the source acquired, is an open call and is not settled by this note.
• van Antwerpen et al. (2016), Statix (via HOAG r2 §B2) — the two-stratum discipline that makes reading an under-construction scope graph sound. gen-pipe realizes the consumer side of that condition, not the resolution calculus.
• Cheney, Chiticariu & Tan (2009), "Provenance in Databases: Why, How, and Where" — the provenance chain is lineage/where-provenance-shaped. gen-pipe does not implement the semiring-annotated generality of Green–Karvounarakis–Tannen (2007); the survey is cited, not the semiring framework, to keep the claim honest.
• Bird & Meertens (1987), the Bird–Meertens formalism — informed-by only. map/fold/scan are the structured list-recursion operators (each with a fusion law and a value-independent output shape), so their contribution count stays readable without forcing values. over (f : [a] → [b]) is the unstructured general list function they specialize — the whole-list escape hatch (reverse/take/cross-element rewrite). It carries no fusion law and its output cardinality depends on the values, so it is value-demanding and strict where fold/scan stay lazy.

The producer-class deferral semantics trace to engineering precedent (den v1 PR #623, HOAG r2's pipe.withConfig thunk contract), cited as design provenance.