Goal
Scaffold the repo and implement the core types. Architecture was settled 2026-07-31 — build to the spec, don't redesign.
Spec
Job(id: str, stage: str, shard: int, inputs: list[str], output: str)
Result(job_id: str, ok: bool, output: str | None, error: str | None, duration: float)
Runner.run(jobs: list[Job]) -> list[Result] # Protocol; sync, takes a whole wave
Jobs cross process boundaries. Job must be JSON-serializable and communicate only through paths — never live Python objects. LocalRunner doesn't need this; Slurm (W5) and K8s (W6) do, and designing for it now is what makes them drop-in.
Runner.run is sync and takes a list because that is how every real backend works — a Slurm job array, a K8s Job with parallelism: N. Submit the wave, wait, collect. Async stays an implementation detail inside a stage.
A single job must be runnable on its own. Slurm launches an array element and K8s runs a
Job pod — neither runs minipipe run; each runs one job from a spec. So minipipe worker <spec.json> is part of the contract, not a convenience. Stub it here; it dispatches for real
in #3.
RunConfig (run_id, shards, level, top) is written once to runs/<run_id>/config.json at submit time; every stage loads and validates it. No per-stage payload models until stages genuinely diverge.
Done when: minipipe imports, mypy strict is clean, and LocalRunner executes a trivial job list.
Goal
Scaffold the repo and implement the core types. Architecture was settled 2026-07-31 — build to the spec, don't redesign.
pyproject, mypy strict, ruff, pytest, CI workflowlogstatsas a uv git dependency (track branch during Week 3; pin a tag at the checkpoint)Job,Result,RunConfigas pydantic modelsRunnerProtocolLocalRunner(sequential, in-process)workloads/logs.pycreated — the only module allowed to import logstats.gitignore:data/(generated logs are never committed)minipipe runandminipipe worker <spec.json>Spec
Jobs cross process boundaries.
Jobmust be JSON-serializable and communicate only through paths — never live Python objects.LocalRunnerdoesn't need this; Slurm (W5) and K8s (W6) do, and designing for it now is what makes them drop-in.Runner.runis sync and takes a list because that is how every real backend works — a Slurm job array, a K8s Job withparallelism: N. Submit the wave, wait, collect. Async stays an implementation detail inside a stage.A single job must be runnable on its own. Slurm launches an array element and K8s runs a
Job pod — neither runs
minipipe run; each runs one job from a spec. Sominipipe worker <spec.json>is part of the contract, not a convenience. Stub it here; it dispatches for realin #3.
RunConfig(run_id, shards, level, top) is written once toruns/<run_id>/config.jsonat submit time; every stage loads and validates it. No per-stage payload models until stages genuinely diverge.Done when:
minipipeimports, mypy strict is clean, andLocalRunnerexecutes a trivial job list.